A micro-expression feature extraction method based on motion unit prototype template

Abstract

The application discloses a micro-expression feature extraction method based on a motion unit prototype template, and comprises the following steps: in the preprocessing, in view of the face difference problems among different subjects and the head translation problems of the same subject in the video, an organ-based face position correction and cutting method is used to ensure the accuracy of the optical flow feature extraction; in order to accurately analyze the motion unit in the micro-expression, a motion unit-oriented prototype template is proposed. The representative face motion unit dynamic information is recorded in each prototype template, and the AU recognition of the face action with certain difference has strong robustness; in order to accurately capture the micro-expression action, the optical flow map sequence of the micro-expression video is matched with the AU motion template, the complex micro-expression action in the video is deeply analyzed, and the accuracy and the explainability of the micro-expression feature are improved. The application has the advantages that the subtle face action when the micro-expression occurs can be effectively captured, and the application can be used for micro-expression detection, recognition and generation.

Description

Technical Field

[0001] This invention relates to the field of micro-expression feature extraction technology, and in particular to a method for micro-expression feature extraction based on a motion unit prototype template. Background Technology

[0002] Facial expressions are an important way for humans to interact emotionally. Based on the duration and intensity of changes in facial emotional states, expressions can be divided into macro-expressions and micro-expressions. Macro-expressions last from 0.5 to 4 seconds and are accompanied by dramatic changes in facial muscles. Micro-expressions are involuntary, rapid expressions, typically lasting less than 0.5 seconds. Neurophysiological studies have shown that micro-expressions are difficult to control consciously; they are genuine expressions of human emotion and serve as a window into the study of cognitive psychology and neural reflexes. Because micro-expression movements are very subtle and short-lived, and are influenced by many factors such as head movements, blinking, and differences in the facial features of the subjects being tested, the accuracy of micro-expression detection and recognition has always been low. Eliminating irrelevant interference and accurately capturing the occurrence of micro-expressions remains a very challenging task.

[0003] Existing micro-expression feature extraction techniques fall into two categories: traditional handcrafted features and deep features. Traditional handcrafted features include texture features and optical flow features. Texture features are highly versatile in image processing, but they are poor at detecting differences in facial changes and struggle to reflect the dynamic evolution of micro-expressions. Optical flow features possess the technical characteristics to describe object motion information and are robust to facial differences, making them ideal for detecting subtle movements like micro-expressions. Researchers have made optical flow methods a key approach in micro-expression research. However, extracting optical flow features requires relying on experience to define facial regions associated with micro-expression movements, which is often not optimal. In recent years, deep learning has achieved success in macro-expression analysis, but the limited size of existing micro-expression databases makes it difficult to meet the requirements of deep learning models, often leading to overfitting and other problems. Furthermore, deep learning-based micro-expression feature extraction methods often lack clear evidence and interpretable results. In applications such as abnormal behavior detection and abnormal emotion monitoring, micro-expression analysis results without clear evidence and a well-defined process are difficult to accept.

[0004] Ekman first proposed the concept of facial motion units (AUs) in facial coding systems. AUs are basic facial movements that can be combined to form various facial expressions. When a micro-expression occurs, one or more motion units will be activated. Facial motion units are a powerful tool for analyzing micro-expressions and have therefore become a hot topic in the field of micro-expression research. Each AU is generated by the traction of fixed facial muscles; therefore, when the same motion unit is activated, similar movement patterns will appear on the face.

[0005] However, due to natural facial differences, different subjects will exhibit varying facial movements when generating the same AU. Even among the same subjects, differences in the intensity of emotion can lead to significant variations in the amplitude of facial movements belonging to the same AU. This poses a challenge to micro-expression AU recognition. Therefore, designing a method that can accurately acquire AU motion patterns and is robust to recognizing facial movements belonging to the same AU is crucial. This not only helps improve the effectiveness of micro-expression detection but also has significant implications for subsequent, more accurate, and interpretable micro-expression modeling. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method for extracting micro-expression features based on motion unit prototype templates.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for extracting micro-expression features based on motion unit prototype templates includes the following steps:

[0009] Step 1: For the training set, select the initial frame and peak frame from the micro-expression videos. The initial frame is the first frame where the micro-expression begins, and the peak frame is the frame with the largest amplitude of the micro-expression movement, representing the greatest motion information.

[0010] Step 2: Locate the coordinates of the feature points in the initial frame, and select the nose tip coordinates from them. Coordinates of the left outer corner of the eye Coordinates of the outer corner of the right eye and the left corner of the mouth Right corner of the mouth

[0011] Step 3: Correct the position of feature points in the peak frame based on the optical flow of the nose tip region in the initial frame and the peak frame.

[0012] Step 4: Cut out the eye area based on the feature points at the corners of the eyes; cut out the mouth area based on the feature points at the corners of the mouth.

[0013] Step 5: Calculate the dense optical flow map of the eye and mouth regions between the initial frame and the peak frame in the micro-expression video i using the dense optical flow method, denoted as . and The formula is as follows:

[0014]

[0015]

[0016] Step 6: Overlay and average the optical flow maps of micro-expression videos with the same motion units to obtain the motion unit prototype template M. jIt represents the basic movement pattern of the corresponding motor unit when a micro-expression occurs; j is the index of the motor unit, H eye It is a collection of motor units that occur in the eye region, H mth It is a collection of motion units occurring in the mouth region; the use of superposition averaging method can effectively suppress random noise in optical flow maps.

[0017]

[0018] S AUj It is a collection of micro-expression video indexes containing motion unit j.

[0019] After calculating all motion unit prototype templates, the motion unit template set mask = {M1,...,M} was obtained. V V is the number of motion units.

[0020] Step 7: For the test set, after preprocessing in steps 2, 3, and 4, use the dense optical flow method to calculate the dense optical flow map of the eye and mouth regions between the two micro-expression images, denoted as F. mth and F eye .

[0021] Step 8: Calculate the optical flow diagram and the motion unit prototype template M j matching degree p j .

[0022]

[0023]

[0024] p j The expression represents the matching result between the optical flow diagram and the prototype template j, where w and h are matrices D. j The width and length.

[0025] Step 9: After matching with all motion unit prototype templates, obtain the micro-expression features P = {p} based on the motion unit templates. 1 ,...,p V}

[0026] Furthermore, the specific sub-steps of step 3 are as follows:

[0027] Step 3-1: Based on the nose tip coordinates of the initial frame Determine the nose tip region bounding boxes NOSE1 and NOSE in the initial frame and peak frame. apex .

[0028] Step 3-2: Calculate the dense optical flow in the nose tip region of the initial and peak frames in the image sequence using the Farneback optical flow method, as shown in the following equation:

[0029] F nose =Farneback(NOSE1,NOSE) apex )

[0030] Obtain the optical flow matrix w nose and h nose These are the width and length of the nasal tip area, respectively.

[0031] Step 3-3: Calculate the optical flow matrix F of the nasal tip region nose mean To measure the change in facial position between two frames, where This represents horizontal movement. Represents movement in the vertical direction.

[0032]

[0033] Step 3-4: Based on the offset of the initial frame and the peak frame obtained in Step 3-3 The feature point coordinates of the peak frame are calculated from the feature point coordinates of the initial frame.

[0034]

[0035]

[0036]

[0037]

[0038] Furthermore, the specific sub-steps of step 4 are as follows:

[0039] Step 4-1: Based on the coordinates of the left outer corner of the eye Coordinates of the outer corner of the right eye Calculate the cut box for the eye region, with the top left corner of the cut box. and bottom right corner Obtain the eye region EYE in the first frame first Similarly, the peak frame EYE is obtained. apex The formula is as follows:

[0040]

[0041]

[0042] Step 4-2: Based on the left corner of the mouth Right corner of the mouth Calculate the cut box for the mouth region, with the top left corner of the cut box as the cut box. and bottom right corner Get the mouth region MTH in the first framefirst Similarly, the peak frame MTH is obtained. apex The formula is as follows:

[0043]

[0044]

[0045] Compared with the prior art, the advantages of the present invention are as follows:

[0046] (1) In the preprocessing, an organ-based facial position correction and cropping method was used to address the issues of facial differences between different subjects and head translation of the same subject in the video, ensuring the accuracy of optical flow feature extraction. (2) To accurately analyze the motion units in micro-expressions, a prototype template oriented towards motion units was proposed. Each prototype template records representative dynamic information of facial motion units and has strong robustness to AU recognition of facial movements with certain differences. (3) To accurately capture micro-expression movements, the optical flow map sequence of the micro-expression video was matched with the AU motion template to obtain an in-depth analysis of complex micro-expression movements in the video, greatly improving the accuracy and interpretability of micro-expression detection. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the calculation of the motion unit prototype template in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram illustrating image preprocessing in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the prototype template corresponding to motion unit 4 (frowning action) in an embodiment of the present invention.

[0050] Figure 4 This is a flowchart illustrating the overall process of the micro-expression detection method based on a motion unit prototype template in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0052] A micro-expression feature extraction method based on motion unit prototype templates is proposed to detect the timing of micro-expressions in long videos. It mainly consists of two parts: calculating the micro-expression motion unit prototype template and detecting micro-expressions based on the matching results between the long video and the motion unit prototype template.

[0053] I. For example Figure 1 As shown, calculating the micro-expression motion unit prototype template includes the following steps:

[0054] Step 1: Based on the database annotations, select the initial frame and peak frame from the micro-expression video. The initial frame is the first frame where the micro-expression begins, and the peak frame is the frame with the largest amplitude of the micro-expression movement, representing the greatest motion information.

[0055] Step 2: Locate the 68 feature points of the initial frame, such as... Figure 2 As shown. Select the coordinates of the nose tip. Coordinates of the left outer corner of the eye Coordinates of the outer corner of the right eye and the left corner of the mouth Right corner of the mouth

[0056] Step 3: Perform facial alignment based on the optical flow of the nose tip region in the initial frame and peak frame;

[0057] Step 3-1: Based on the nose tip coordinates p of the initial frame nose =(x nose ,y nose ), determine the nose tip region bounding boxes NOSE1 and NOSE in the initial frame and peak frame. apex .

[0058] Step 3-2: Calculate the dense optical flow in the nose tip region of the initial and peak frames in the image sequence using the Farneback optical flow method, as shown in the following equation:

[0059] F nose =Farneback(NOSE1,NOSE) apex )

[0060] Obtain the optical flow matrix w nose and h nose These are the width and length of the nasal tip area, respectively.

[0061] Step 3-3: Calculate the optical flow matrix F of the nasal tip region nose mean To measure the change in facial position between two frames, where This represents horizontal movement. Represents movement in the vertical direction.

[0062]

[0063] Step 3-4: Based on the offset of the initial frame and the peak frame obtained in Step 3-3 The feature point coordinates of the peak frame are calculated from the feature point coordinates of the initial frame.

[0064]

[0065]

[0066]

[0067]

[0068] Step 4: Calculate the eye and mouth regions based on the coordinates of the initial and peak frames, such as... Figure 2 As shown.

[0069] Step 4-1: Taking the calculation of the first frame as an example, based on the coordinates of the left outer corner of the eye. Coordinates of the outer corner of the right eye Calculate the cut box for the eye region, with the top left corner of the cut box. and bottom right corner Get the eye area EYE first As shown in the formula below. Similar to processing peak frames, the EYE is obtained. apex .

[0070]

[0071]

[0072] Step 4-2: Taking the calculation of the first frame as an example, based on the left corner of the mouth Right corner of the mouth Calculate the cut box for the mouth region, with the top left corner of the cut box as the cut box. and bottom right corner Get the mouth area MTH first Similar to processing peak frames, we obtain the MTH. apex .

[0073]

[0074]

[0075] Step 5: Calculate the dense optical flow map of the eye and mouth regions between the initial frame and the peak frame in the micro-expression video i using the dense optical flow method, denoted as . and

[0076]

[0077]

[0078] Step 6: Overlay and average the optical flow maps of micro-expression videos with the same motion units to obtain the motion unit prototype template M. j It represents the basic movement pattern of the corresponding motor units when micro-expressions occur, such as... Figure 3 As shown; j is the index of the motion unit.

[0079]

[0080] H eye It is a collection of motor units that occur in the eye region, H mth It is a collection of motor units that occur in the mouth region. S AUj It is a set of micro-expression video indexes containing motion unit j. After calculating all motion unit prototype templates, the motion unit prototype template set mask = {M1,...,M} is obtained. V}; V is the number of motion units.

[0081] II. Figure 4 As shown, micro-expression features are calculated based on micro-expression motion unit prototype templates to achieve micro-expression detection in long videos, including the following steps:

[0082] Step 1: Divide the micro-expression video into frames, and then form a micro-expression sequence I = {I1, I2, ..., I...} by sequentially analyzing each frame. m I1 is used as a reference frame for micro-expression movements in the video.

[0083] Step 2: Locate the coordinates of the tip of the nose (p) in I1. nose =(x nose ,y nose ), coordinates of the left outer corner of the eye p leye =(x leye ,y leye ), coordinates of the right outer corner of the eye p reye =(x reye ,y reye ) and the left corner of the mouth lm right corner of the mouth p rm The coordinates.

[0084] Step 3: Correct the facial position in the video based on the optical flow of the nose tip area.

[0085] Step 3-1: Based on the nose tip coordinates p of the initial frame nose =(x nose ,y nose Determine the nose tip region bounding boxes NOSE1 and NOSE in the initial frame and the current frame. c .

[0086] Step 3-2: Calculate the dense optical flow in the nose tip region of the image sequence in the initial and current frames using the Farneback optical flow method.

[0087] F nose =Farneback(NOSE1,NOSE) c )

[0088] Obtain the optical flow matrix w nose and h nose These are the width and length of the nasal tip area, respectively.

[0089] Step 3-3: Calculate the optical flow matrix F of the nasal tip region nose mean To measure the change in facial position between two frames, where This represents horizontal movement. Represents movement in the vertical direction.

[0090]

[0091] Step 3-4: Based on the offset obtained in Step 3-3 Calculate the feature point coordinates of the current frame using the feature point coordinates of the initial frame.

[0092]

[0093]

[0094]

[0095]

[0096] Step 4: Calculate the eye and mouth regions based on the coordinates of the initial frame, such as... Figure 2 As shown.

[0097] Step 4-1: Taking the calculation of the first frame as an example, based on the coordinates of the left outer corner of the eye. Coordinates of the outer corner of the right eye Calculate the cut box for the eye region, with the top left corner of the cut box. and bottom right corner Get the eye area EYE first As shown in the formula below. Similar to processing the current frame, we obtain EYE. c .

[0098]

[0099]

[0100]

[0101]

[0102] Step 4-2: Taking the calculation of the first frame as an example, based on the left corner of the mouth Right corner of the mouth Calculate the cut box for the mouth region, with the top left corner of the cut box as the cut box. and bottom right corner Obtain the mouth region MTH1. Process the current frame similarly to obtain MTH. c .

[0103]

[0104]

[0105] Step 5: Calculate the dense optical flow map of the eye and mouth regions between the initial frame and the current frame using the dense optical flow method, denoted as [image of map]. and

[0106]

[0107]

[0108] Step 6: Using the first image as a reference, perform image processing on the image sequence I = {I1, I2, ..., I...} m Steps 3, 4, and 5 are performed on each frame in the sequence and reference frame I1 to obtain the optical flow image sequence of the eye region. Optical flow image sequence of the mouth region

[0109] Step 7: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] and The optical flow features in the graph are compared with the prototype template of the moving unit, as shown in the formula below, to calculate the i-th optical flow graph. and With the motion unit prototype template M j degree of matching

[0110]

[0111]

[0112] This represents the matching result between the i-th optical flow map and the prototype template j. Optical flow maps with a high degree of matching indicate that there are motion units in the micro-expression video that correspond to the prototype template.

[0113] Step 8: Therefore, by comparing all optical flow maps in the window with V prototype templates, a V*(N-1) feature matrix W can be obtained to describe the occurrence of various motion units in the micro-expression image sequence in the window.

[0114]

[0115] Step 9: The feature matrix W contains V time-domain variation curves W = [L1, L2, ..., L VWhen facial expressions occur, there are obvious waveform changes in the waveform curves of the corresponding several motion units.

[0116] Step 10: Using a peak localization method, based on the time-domain variation curve L j The waveform changes determine the occurrence time of the motion unit, and the corresponding characteristic curve L is obtained. j Time pairs of set T j T j ={[start1,end1] j ,......,[start n end n ] j} represents the calculated start and end times of the nth motion unit. This method mainly consists of three steps:

[0117] Step 10-1: Use a low-pass filter to remove high-frequency fluctuations in the curve caused by noise.

[0118] Step 10-2: Use a sliding window of appropriate size to find the peak. When the curve value at the center of the sliding window exceeds a certain threshold compared to the minimum values ​​on both sides of the center, the center point of the sliding window is considered to be a peak. Continue moving the sliding window backward to calculate the new center point.

[0119] Step 10-3: Step 10-2 only found the peak of the wave, that is, the few frames near the peak frame. In order to obtain the complete timing of the micro-expression, it is necessary to extend to the left and right of the peak until the relatively flat feature curve part.

[0120] Step 11: Perform step 9 on each time-domain variation curve to obtain the sequence {T1,T2,...,T...} V}. Perform the union operation on all sets in the sequence to obtain T = {T1∪T2∪...∪T}. V}

[0121] Step 12: Calculate the two time pairs belonging to different motion units within time T [start] n end n ] and [start m end m The intersection index r of ] .

[0122]

[0123] Step 13: If the intersection index r is greater than the threshold θ, then the two AU actions are considered to belong to the same micro-expression. Calculate the new time pair [start] new end new ].

[0124] start new =min(start) n ,start m ),end new =max(end) n end m )

[0125] Remove [start] from T n end n ] and [start m end m ], and insert [start] new end new ].

[0126] Step 14: After performing steps 12 and 13 on all two time pairs belonging to different motion units within T, the set of time pairs T is the result of micro-expression detection.

[0127] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A method for extracting micro-expression features based on motion unit prototype templates, characterized in that, Includes the following steps: Step 1: For the training set, select the initial frame and peak frame from the micro-expression videos; the initial frame is the first frame where the micro-expression begins, and the peak frame is the frame with the largest amplitude of the micro-expression movement, representing the greatest motion information; Step 2: Locate the coordinates of the feature points in the initial frame, and select the nose tip coordinates from them. =( Coordinates of the left outer corner of the eye =( ), coordinates of the right outer corner of the eye =( ) and the left corner of the mouth right corner of the mouth ; Step 3: Correct the position of feature points in the peak frame based on the optical flow of the nose tip region in the initial frame and the peak frame; Step 3 consists of the following sub-steps: Step 3-1: Based on the nose tip coordinates of the initial frame =( Determine the nose tip region bounding box in the initial frame and peak frame. and ; Step 3-2: Calculate the dense optical flow in the nose tip region of the initial and peak frames in the image sequence using the Farneback optical flow method, as shown in the following equation: ， Obtain the optical flow matrix , and These are the width and length of the nasal tip area, respectively. Step 3-3: Calculate the optical flow matrix of the nasal tip region mean To measure the change in facial position between two frames, where This represents horizontal movement. Represents movement in the vertical direction; ， Step 3-4: Based on the offset of the initial frame and the peak frame obtained in Step 3-3 Calculate the feature point coordinates of the peak frame using the feature point coordinates of the initial frame; ， ， ， ， Step 4: Cut out the eye area based on the feature points at the corners of the eyes; cut out the mouth area based on the feature points at the corners of the mouth; Step 5: Calculate the dense optical flow map of the eye and mouth regions between the initial frame and the peak frame in the micro-expression video i using the dense optical flow method, denoted as . and The formula is as follows: ， ， Step 6: Overlay and average the optical flow maps of micro-expression videos with the same motion units to obtain the motion unit prototype template. It represents the basic movement pattern of the corresponding motor unit when a micro-expression occurs; j is the index of the motor unit. It is a collection of motor units that occur in the eye region. It is a collection of motor units that occur in the mouth region; ， It is a collection of micro-expression video indexes containing motion unit j; After calculating all motion unit prototype templates, a set of motion unit templates was obtained. V is the number of motion units; Step 7: For the test set, after preprocessing in steps 2, 3, and 4, use the dense optical flow method to calculate the dense optical flow map of the eye and mouth regions between the two micro-expression images, denoted as . and ; Step 8: Calculate the optical flow diagram and motion unit prototype template degree of matching ; ， ， Representing optical flow diagrams and prototype templates The matching results, where w and h are matrices. Width and length; Step 9: After matching with all motion unit prototype templates, micro-expression features based on motion unit templates are obtained. .

2. The method for extracting micro-expression features based on a motion unit prototype template according to claim 1, characterized in that: Step 4 consists of the following sub-steps: Step 4-1: Based on the coordinates of the left outer corner of the eye =( ), coordinates of the right outer corner of the eye =( Calculate the cut box for the eye region, with the top left corner of the cut box... =( ) and bottom right corner =( ), thus obtaining the first frame's eye region ; The formula is as follows: ， ， ， Step 4-2: Based on the left corner of the mouth right corner of the mouth Calculate the cut box for the mouth region, with the top left corner of the cut box as the cut box. =( ) and bottom right corner =( ), thus obtaining the mouth region in the first frame. ; The formula is as follows: ， 。

Images