A robust micro-expression segment segmentation method and system that circumvents head motion
By using computer vision tools to identify head poses and segment video segments with head movement amplitudes less than a threshold, the problem of low accuracy in micro-expression detection caused by head movement interference is solved, and efficient micro-expression sample segmentation and detection is achieved in non-laboratory environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF PSYCHOLOGY CHINESE ACADEMY OF SCI
- Filing Date
- 2023-07-08
- Publication Date
- 2026-05-29
AI Technical Summary
Existing micro-expression detection methods are affected by head movements in non-laboratory environments, resulting in low detection accuracy and recall. Furthermore, deep learning methods are affected by small sample size issues, making them difficult to apply in real-world scenarios.
Computer vision tools are used to identify head poses, the amplitude of head movement is calculated by frame difference method, video segments with head movement amplitude less than a set threshold are segmented, and these segments are aligned to segment micro-expression samples, thus constructing a micro-expression segment segmentation system.
It effectively avoids head movement interference, improves the performance of micro-expression detection, alleviates the small sample problem, is suitable for various application scenarios, and has an efficient, intuitive and highly accurate micro-expression sample segmentation method.
Smart Images
Figure CN116863140B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of micro-expression technology, specifically relating to a robust micro-expression segmentation method and system that avoids head movements. Background Technology
[0002] Microexpression research has significant application potential in psychology and human-computer interaction, particularly in emotion recognition, lie detection, and emotional intelligence, thanks to its ability to provide clues to reveal inner states and the inherently difficult-to-suppress nature of microexpressions. However, head movement is a common interfering factor in microexpression detection. Since microexpressions are brief and minute muscle movements in human facial expressions, they are easily obscured by the vast amount of head movement, distorting the temporal and spatial characteristics of microexpressions. This can lead to lower accuracy and recall (typically measured by the F1-score) in microexpression detection methods. Given the short duration (less than 0.5 seconds) and low intensity of microexpressions, most existing microexpression detection methods are based on laboratory environments. Data collection in laboratory settings requires researchers to keep their heads as still as possible. However, current deep learning-based microexpression detection methods are heavily influenced by small sample sizes, hindering their application in real-world scenarios. Summary of the Invention
[0003] To avoid the influence of head posture movements on micro-expressions, this invention provides a robust micro-expression segmentation method and system that avoids head movements. It segments samples for micro-expression annotation and detection from videos collected in non-laboratory environments, thereby alleviating the problem of small sample size in micro-expression detection.
[0004] The specific plan adopted is as follows:
[0005] On the one hand, the present invention provides a robust micro-expression segmentation method that avoids head movements. It uses computer vision tools to perform head pose recognition on each frame of the video sample and extracts the translation and rotation variables of the head pose relative to the same coordinate axis. It uses the frame difference method to calculate the head movement amplitude and divides the video sample into several consecutive frame intervals where the head movement amplitude is less than a set displacement threshold. After adjusting and aligning each frame of the video sample with consecutive frame intervals, the micro-expression sample is obtained.
[0006] Preferably, the method of calculating head movement amplitude using the frame difference method, dividing the video samples into several consecutive frame intervals where the head movement amplitude is less than a set displacement threshold, is specifically as follows:
[0007] Calculate the displacement distance d1 of the translation variable and the displacement distance d2 of the rotation variable between adjacent frames according to the playback time sequence of the video samples.
[0008] The obtained displacement distance d1 is compared with the set displacement threshold τ1 in turn, and the obtained displacement distance d2 is compared with the set displacement threshold τ2 in turn.
[0009] Collect all consecutive frame intervals that satisfy the condition that distance d1 is less than displacement threshold τ1 and displacement distance d2 is less than displacement distance τ2, and segment them from the entire video sample.
[0010] Or preferably, the method of calculating head movement amplitude using the frame difference method, dividing the video samples into several frame intervals where the head movement amplitude is less than a set displacement threshold, is specifically as follows:
[0011] Calculate the translation distance d1' and rotation distance d2' between each frame and the starting frame according to the playback time sequence of the video samples;
[0012] The obtained displacement distance d1' is compared with the set displacement threshold τ1 in sequence, and the obtained displacement distance d2' is compared with the set displacement distance τ2 in sequence;
[0013] Collect all consecutive frame intervals that satisfy the condition that distance d1' is less than displacement threshold τ1 and displacement distance d2' is less than displacement distance τ2, and segment them from the entire video sample.
[0014] Furthermore, the displacement distances d1 and d1' are calculated using the following Euclidean distance:
[0015]
[0016] The displacement distances d2 and d2' are calculated using the following cosine distance:
[0017]
[0018] Where: d1(A,B) and d1′(A,B) represent the Euclidean distance between vectors A and B of the two frames;
[0019] d2(A,B) and d2′(A,B) represent the cosine distance between vectors A and B of two frames of images;
[0020] n represents that the vector is in an n-dimensional space, where i∈[x,y,z].
[0021] Furthermore, if there exists a distance between adjacent frames in a continuous frame interval that satisfies d1(A,B)>τ1 and d2(A,B)>τ2, or if the difference in the number of frames between d1(A,B)>τ1 and d2(A,B)>τ2 is less than 0.5*fps, it is determined that this frame image has a large-scale head movement. The video sample is divided at this point, and this frame is used as a new starting frame for subsequent calculations and comparisons between adjacent frames to select the continuous frame interval that meets the requirements.
[0022] Alternatively, if there exists a continuous frame interval where the distance between the starting frame and each subsequent frame satisfies d1′(A,B)>τ1 and d2′(A,B)>τ2, or if the difference in the number of frames between d1′(A,B)>τ1 and d2′(A,B)>τ2 is less than 0.5*fps, then it is determined that this frame image has a large-scale head movement. The video sample is divided at this point, and this frame is used as the new starting frame for calculation and comparison between each subsequent frame and the new starting frame, selecting the continuous frame interval that meets the requirements.
[0023] Furthermore, the extraction of head pose information using computer vision tools specifically includes: detecting that the facial confidence level of the person's head is greater than 95%, and obtaining the translation and rotation variable information of the person's head.
[0024] On the other hand, the present invention also provides a robust micro-expression segmentation system that avoids head movements, the system running in a computer and comprising:
[0025] The video sample library is used to store user video sample images collected;
[0026] The visual tool uses the open-source toolkit dlib to extract video samples from the video sample library and identifies the head pose of each frame in the video sample, extracting the translation and rotation variables of the head pose relative to the same coordinate axis.
[0027] The calculation module extracts the translation and rotation variables of each frame in the order of video playback time, and uses the frame difference method to calculate the head movement amplitude d;
[0028] The comparison module is equipped with a displacement threshold τ, which is used to compare the numerical values between the head movement amplitude d and the displacement threshold τ.
[0029] The video segmentation module is used to extract continuous frame intervals from video samples when the head movement amplitude d is less than the displacement threshold τ.
[0030] The micro-expression processing module is used to align the continuous frame interval images captured by the video segmentation module.
[0031] A micro-expression sample library is used to store several micro-expression samples after adjustment and alignment.
[0032] Furthermore, the calculation module includes a calculation module I, which is used to calculate the displacement distance d1 of the translation variable and the displacement distance d2 of the rotation variable between adjacent frames; the comparison module sets displacement thresholds τ1 and τ2, and the comparison module sequentially compares the obtained displacement distance d1 with the set displacement threshold τ1, and compares the obtained displacement distance d2 with the set displacement threshold τ2.
[0033] Furthermore, the calculation module also includes a calculation module II, which is used to calculate the translation distance d1' and rotation distance d2' between each frame image and the starting frame image, respectively. The comparison module sets displacement thresholds τ1 and τ2, and the comparison module sequentially compares the obtained displacement distance d1' with the set displacement threshold τ1 and compares the obtained displacement distance d2' with the set displacement distance τ2.
[0034] The technical solution of the present invention has the following advantages:
[0035] A. The micro-expression segmentation method and system provided by the present invention calculates the head movement amplitude between frames and segments usable small-amplitude head movement video segments as micro-expression samples. This can avoid the influence of head posture on micro-expression detection. Within a certain displacement threshold angle, the influence of head movement can be eliminated by aligning or removing global motion, thereby improving the performance of micro-expression detection.
[0036] B. This invention is not limited to any particular application scenario. It can segment videos with small head movements into usable micro-expression samples, thereby alleviating the problem of insufficient micro-expression dataset samples to a certain extent and providing a foundation for applying advanced technologies such as deep learning to micro-expression detection.
[0037] C. The avoidance method proposed in this invention can estimate head movement in a video in real time, and it does not require data for training. It has the advantages of high efficiency, intuitiveness, high accuracy and stability. Attached Figure Description
[0038] To more clearly illustrate the specific embodiments of the present invention, the accompanying drawings used in the specific embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0039] Figure 1 The present invention provides head pose data for extracting each frame of a video sample using computer vision tools.
[0040] Figure 2 This is an illustration of a video sample for calculating cosine distance and Euclidean distance provided by the present invention;
[0041] Figure 3 This is a structural diagram of the robust micro-expression segmentation system that avoids head movements provided by the present invention. Detailed Implementation
[0042] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] like Figure 1 As shown, this invention provides a robust micro-expression segmentation method that avoids head movements. The specific method is as follows:
[0044]
S01
[0045] This invention employs computer vision tools to extract human head pose information, specifically in two steps. First, the confidence level of the detected face must be greater than 95% to ensure that the tracked target belongs to the same person. Second, this invention only needs to acquire head translation and rotation information, resulting in faster computation. The specific process is as follows: Figure 1 As shown, the translation variables (pose_Tx, pose_Ty, pose_Tz) of the head pose are extracted from each frame of the image. These variables represent the head's position relative to the camera. The rotation variables (pose_Rx, pose_Ry, pose_Rz) represent the rotation of the head position relative to the X, Y, and Z axes, respectively. By convention, the left hand is positive, and these can be regarded as pitch (Rx), yaw (Ry), and roll (Rz). The rotation variables are with the camera as the origin of the coordinate axis. Figure 1 In this context, Orientation represents the rotation angle, and Pose represents the relative position from the camera.
[0046]
S02
[0047] Specifically, this invention employs two methods for calculating head movement amplitude:
[0048] The first calculation method involves calculating the displacement distance between translation and rotation variables of adjacent frames, and then selecting an appropriate displacement distance threshold through debugging to segment the video samples.
[0049] First, calculate the displacement distance d1 of the translation variable and the displacement distance d2 of the rotation variable between adjacent frames according to the playback time order of the video samples;
[0050] Next, the obtained displacement distance d1 is compared with the set displacement threshold τ1, and the obtained displacement distance d2 is compared with the set displacement threshold τ2.
[0051] Finally, all consecutive frame intervals that satisfy the condition that distance d1 is less than displacement threshold τ1 and displacement distance d2 is less than displacement distance τ2 are collected and segmented from the entire video sample.
[0052] The second calculation method is to segment the video by calculating the displacement distance between the head pose in each frame and the starting frame, and then adjusting the threshold.
[0053] First, calculate the translation distance d1' and rotation distance d2' between each frame and the starting frame according to the playback time sequence of the video samples;
[0054] Next, the obtained displacement distance d1' is compared with the set displacement threshold τ1, and the obtained displacement distance d2' is compared with the set displacement distance τ2.
[0055] Finally, all consecutive frame intervals that satisfy the condition that distance d1' is less than displacement threshold τ1 and displacement distance d2' is less than displacement distance τ2 are collected and segmented from the entire video sample.
[0056] The displacement distances d1 and d1' mentioned above are calculated using the following Euclidean distances:
[0057]
[0058] The displacement distances d2 and d2' mentioned above are calculated using the following cosine distance:
[0059]
[0060] Where: d1(A,B) and d1′(A,B) represent the Euclidean distance between vectors A and B of the two frames;
[0061] d2(A,B) and d2′(A,B) represent the cosine distance between vectors A and B of two frames of images;
[0062] n represents that the vector is in an n-dimensional space, where i∈[x,y,z].
[0063] Since cosine distance is better at processing directional information between vectors, this invention can achieve more accurate displacement calculation by calculating the Euclidean distance between translation vectors and the cosine distance between rotation variables.
[0064] If the Euclidean distance between frames is less than the threshold τ1, it is considered that the translational change of the head posture in this continuous frame segment is small, and head movement can be eliminated by scaling for alignment; if the cosine distance between frames is less than the threshold τ2, it is considered that the rotational change of the head posture in this continuous frame segment is small, and alignment can be achieved by rotation matrix.
[0065] If the calculated cosine distance and Euclidean distance exceed the set displacement threshold, the video sample is segmented. The segmented intervals have less head movement influence, and head movement can be eliminated by alignment, thereby improving the performance of micro-expression detection.
[0066] [S03] After adjusting and aligning each frame of the segmented video sample with continuous frame intervals, micro-expression samples are obtained. The multiple video samples obtained after segmentation can be used as samples for micro-expression detection, and then micro-expression annotation can be performed.
[0067] The present invention preferably uses the frame difference method to calculate the head movement amplitude. The video sample is divided into several frame intervals with small head movement amplitude. These images with continuous frame intervals can be applied to micro-expression annotation and detection.
[0068] When calculating the distance between adjacent frames using the first calculation method, this invention calculates the Euclidean distance d1(A,B) between the translation variables (pose_Tx, pose_Ty, pose_Tz) between adjacent frames frame by frame, and calculates the cosine distance d2(A,B) between the rotation variables (pose_Rx, pose_Ry, pose_Rz) between adjacent frames. The results are as follows: Figure 2 As shown, the horizontal axis represents the number of video frames, and the vertical axis represents the distance between adjacent frames. If d1(A,B)>τ1 and d2(A,B)>τ2, or when the difference in the number of frames between d1(A,B)>τ1 and d2(A,B)>τ2 is less than 0.5*fps, this invention considers that there is a large amount of head movement at this time, and the video is divided at this point.
[0069] The second calculation method is used to calculate the distance between the head position and the starting frame in each frame: This invention calculates the Euclidean distance d1′(A,B) between the translation variables (pose_Tx, pose_Ty, pose_Tz) between the starting frame and each frame after the starting frame, and the cosine distance d2′(A,B) between the rotation variables (pose_Rx, pose_Ry, pose_Rz) between the starting frame and each frame after the starting frame, where A and B represent the head pose vectors of the two frames, respectively. If d1′(A,B)>τ1 and d2′(A,B)>τ2, or if the difference in the number of frames between d1′(A,B)>τ1 and d2′(A,B)>τ2 is less than 0.5*fps, this invention considers that there is a large amount of head movement, and the video is divided at this point.
[0070] Based on this, the present invention takes into account a sliding window of 0.5*fps. In order to prevent micro-expressions from appearing in areas of intense head movement, the distance of one sliding window is moved forward from the division point as the start of a new segment, and the starting frame is replaced with this frame image as the "new starting frame" for calculation. The distance between the "new starting frame" and each subsequent frame is calculated frame by frame.
[0071] This invention similarly segments the video into several video segments to be detected, and then uses micro-expression detection technology to detect the segments, which can alleviate the problem of time-consuming annotation.
[0072] like Figure 3 As shown, this invention also provides a robust micro-expression segmentation system that avoids head movements. Running on a computer, it includes: a video sample library, a visual tool, a calculation module, a comparison module, a video segmentation module, a micro-expression processing module, and a micro-expression sample library. The video sample library stores user video sample images collected from the computer. The visual tool uses the open-source toolkit dlib to extract video samples from the video sample library and identifies the head posture of each frame in the video samples, extracting the translation and rotation variables of the head posture relative to the same coordinate axis. The calculation module extracts the translation and rotation variables of each frame in the order of video playback time and calculates the head movement amplitude d using the frame difference method. The comparison module has a displacement threshold τ to compare the numerical value between the head movement amplitude d and the displacement threshold τ. The video segmentation module is used to extract continuous frame intervals from the video samples where the head movement amplitude d is less than the displacement threshold τ. The micro-expression processing module is used to align the continuous frame intervals extracted by the video segmentation module. The micro-expression sample library stores several micro-expression samples after adjustment and alignment.
[0073] The calculation module includes a calculation module I, which is used to calculate the displacement distance d1 of the translation variable and the displacement distance d2 of the rotation variable between adjacent frames. The comparison module sets displacement thresholds τ1 and τ2. The comparison module compares the obtained displacement distance d1 with the set displacement threshold τ1 and the obtained displacement distance d2 with the set displacement threshold τ2 in turn.
[0074] Furthermore, the present invention also includes a calculation module II in the calculation module, which differs from the calculation method of calculation module I. Calculation module II is used to calculate the displacement distance d1' of the translation variable and the displacement distance d2' of the rotation variable between each frame image and the starting frame image respectively. The comparison module sets displacement thresholds τ1 and τ2. The comparison module compares the obtained displacement distance d1' with the set displacement threshold τ1 in sequence, and compares the obtained displacement distance d2' with the set displacement distance τ2.
[0075] The displacement thresholds τ1 and τ2 set in calculation module I and calculation module II can be kept consistent. Of course, the displacement thresholds τ1 and τ2 in the two calculation modules can also be kept inconsistent and can be set separately. The details will not be elaborated here.
[0076] In summary, to avoid the influence of head posture movement on micro-expressions, this invention assumes that head posture is a low-dimensional manifold, where each point or vector represents a specific head posture, and there may be some correlation or similarity between adjacent points or vectors. When the correlation of consecutive head postures is less than a set displacement threshold τ, it is considered that the head movement signals within this interval can be aligned, thereby segmenting video segments with relatively fixed positions and rotation angles. Micro-expression detection is then performed on the segmented video segments, thereby improving the accuracy and recall of micro-expression detection.
[0077] This invention enables the application of micro-expression segmentation methods and systems to real-world scenarios. By avoiding the influence of head movements, samples can be segmented from videos collected in non-laboratory environments, thereby alleviating the problem of small sample sizes in micro-expression detection.
[0078] Any aspects not covered in this invention are applicable to the prior art.
[0079] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A robust micro-expression segmentation method that avoids head movements, characterized in that, Computer vision tools were used to perform head pose recognition on each frame of the video samples to obtain human heads with a face confidence greater than 95%, and the translation and rotation variables of the head pose relative to the same coordinate axis were extracted. The frame difference method was used to calculate the head movement amplitude, and the video samples were divided into several continuous frame intervals with head movement amplitude less than a set displacement threshold. After adjusting and aligning each frame of the video samples with continuous frame intervals, micro-expression samples were obtained.
2. The robust micro-expression segmentation method for avoiding head movements according to claim 1, characterized in that, The method described above, which uses frame difference to calculate head movement amplitude and divides video samples into several consecutive frame intervals where the head movement amplitude is less than a set displacement threshold, is specifically as follows: Calculate the displacement distance d1 of the translation variable and the displacement distance d2 of the rotation variable between adjacent frames according to the playback time sequence of the video samples. The obtained displacement distance d1 is compared with the set displacement threshold τ1 in turn, and the obtained displacement distance d2 is compared with the set displacement threshold τ2 in turn. Collect all consecutive frame intervals that satisfy the condition that distance d1 is less than displacement threshold τ1 and displacement distance d2 is less than displacement distance τ2, and segment them from the entire video sample.
3. The robust micro-expression segmentation method for avoiding head movements according to claim 1, characterized in that, The method described above, which uses frame difference to calculate head movement amplitude and divides video samples into several frame intervals where the head movement amplitude is less than a set displacement threshold, is specifically as follows: Calculate the displacement distance d1 of the translation variable between each frame and the starting frame according to the playback time sequence of the video samples. ’ and the displacement distance d2 of the rotational variable ’ ; The obtained displacement distance d1 is then... ’ The obtained displacement distance d2 is compared with the set displacement threshold τ1. ’ Compare with the set displacement distance τ2; Collect items that satisfy distance d1 ’ Less than the displacement threshold τ1 and displacement distance d2 ’ All consecutive frame intervals less than the displacement distance τ2 are segmented from the entire video sample.
4. The robust micro-expression segmentation method for avoiding head movements according to claim 2 or 3, characterized in that, The displacement distances d1 and d1 ’ The following Euclidean distance calculations are used: ； The displacement distances d2 and d2 ’ The following cosine distance is used for calculation: ； Where: d1(A,B) and d1 ’ (A,B) represents the Euclidean distance between vectors A and B of two frames of images; d2(A,B) and d2 ’ (A,B) represents the cosine distance between vectors A and B of two frames; n represents the vector residing in an n-dimensional space. .
5. The robust micro-expression segmentation method for avoiding head movements according to claim 4, characterized in that, If there are adjacent frames in a continuous frame interval whose distances satisfy d1(A,B)>τ1 and d2(A,B)>τ2, or whose difference in the number of frames between d1(A,B)>τ1 and d2(A,B)>τ2 is less than 0.5*fps, then the image of this frame is judged to have a large-scale head movement. The video sample is divided at this point, and this frame is used as the new starting frame for subsequent calculations and comparisons between adjacent frames to select the continuous frame interval that meets the requirements.
6. The robust micro-expression segmentation method for avoiding head movements according to claim 4, characterized in that, If there exists a starting frame in a continuous frame interval and the distance between each frame after the starting frame conforms to d1 ’ (A,B)>τ1 and d2 ’ When (A,B)>τ2 or meets the condition d1 ’ (A,B)>τ1 and d2 ’ When the number of frames between (A,B)>τ2 is less than 0.5*fps, it is determined that there is a large-amplitude head movement in this frame. The video sample is divided at this point, and this frame is used as the new starting frame for calculation and comparison between each subsequent frame and the new starting frame to select the continuous frame interval that meets the requirements.
7. A robust micro-expression segmentation system that avoids head movements, characterized in that, The system runs on a computer and includes: The video sample library is used to store user video sample images collected; The visual tool uses the open-source toolkit dlib to extract video samples from the video sample library and identifies the head pose of each frame in the video sample, extracting the translation and rotation variables of the head pose relative to the same coordinate axis. The calculation module extracts the translation and rotation variables of each frame in the order of video playback time, and uses the frame difference method to calculate the head movement amplitude d; The comparison module is equipped with a displacement threshold τ, which is used to compare the numerical values between the head movement amplitude d and the displacement threshold τ. The video segmentation module is used to extract continuous frame intervals from video samples when the head movement amplitude d is less than the displacement threshold τ. The micro-expression processing module is used to align the continuous frame interval images captured by the video segmentation module. A micro-expression sample library is used to store several micro-expression samples after adjustment and alignment.
8. The robust micro-expression segmentation system for avoiding head movements according to claim 7, characterized in that, The calculation module includes a calculation module I, which is used to calculate the displacement distance d1 of the translation variable and the displacement distance d2 of the rotation variable between adjacent frames. The comparison module sets displacement thresholds τ1 and τ2. The comparison module compares the obtained displacement distance d1 with the set displacement threshold τ1 in sequence, and compares the obtained displacement distance d2 with the set displacement threshold τ2 in sequence.
9. The robust micro-expression segmentation system for avoiding head movements according to claim 7 or 8, characterized in that, The calculation module also includes a calculation module II, which is used to calculate the displacement distance d1 of the translation variable between each frame image and the starting frame image. ’ and the displacement distance d2 of the rotational variable ’ The comparison module sets displacement thresholds τ1 and τ2, and sequentially compares the obtained displacement distances d1 and d2. ’ The obtained displacement distance d2 is compared with the set displacement threshold τ1. ’ Compare with the set displacement distance τ2.