Micro-expression real-time detection method

Abstract

The invention discloses a micro-expression real-time detection method. The method comprises the following steps: acquiring a real-time video; segmenting the video to obtain each frame of face region image, and performing face feature point detection; analyzing a facial muscle action rule, and selecting a key facial region for data input; converting the captured facial region image data into a GPU (Graphics Processing Unit) computing format; performing motion field feature extraction on the converted image data by using a GPU; adopting a feature selection algorithm to screen out a feature value with the highest characterization from the extracted motion field features; performing multi-layer filtering, interval screening and feature fusion on the extracted feature values, and determining a key interval for micro-expression detection; and outputting a micro-expression real-time detection result in combination with the key interval of micro-expression detection. According to the method, a GPU-based motion field feature calculation framework is introduced, the data processing speed and real-time performance of micro-expression detection are improved, the micro-expression feature calculation performance is effectively improved, and real-time micro-expression detection based on motion field features is realized through real-time processing.

Description

Technical Field

[0001] The invention belongs to the technical field of micro-expressions, and in particular relates to a real-time detection method for micro-expressions. Background Art

[0002] Micro-expressions are involuntary, short-lived and weak facial expressions that are often revealed when individuals try to hide their true emotions. Due to the action characteristics of micro-expressions, micro-expression detection faces many challenges. First, micro-expressions often occur only in local areas of the face. Inputting the entire face or inappropriate facial areas may cause the algorithm to be unable to effectively capture the occurrence of micro-expressions. Secondly, current micro-expression detection algorithms often take a lot of time to calculate features. Furthermore, in real interactive scenarios, individual head movements are inevitable. Current algorithms often require the face to be in the front, and micro-expression recognition cannot be performed when the head is greatly deflected. Finally, micro-expression analysis is often used in interactive scenarios of conversation and expression, in which changes in lip shape are an interference factor that cannot be ignored. In particular, changes in lip shape will affect the muscle movements of the lower face and even the face. However, it is difficult to distinguish lip shape changes from facial expressions based on visual signals alone. Summary of the invention

[0003] In order to solve the technical problems existing in the prior art, determine the key facial areas for evaluating and analyzing micro-expressions, and improve the calculation speed of the micro-expression detection algorithm, the present invention provides a real-time micro-expression detection method.

[0004] The specific scheme adopted is as follows:

[0005] A real-time micro-expression detection method, comprising:

[0006] Get real-time video;

[0007] Segment the video, obtain the facial region image of each frame, and perform facial feature point detection;

[0008] Analyze the movement patterns of facial muscles and select key facial areas for data input;

[0009] Convert the captured facial area image data into GPU computing format;

[0010] Use GPU to extract motion field features from the converted image data and select the most representative feature values;

[0011] The extracted feature values ​​are subjected to multi-layer filtering, interval screening and feature fusion to determine the key interval for micro-expression detection;

[0012] Combined with the key intervals of micro-expression detection, the real-time detection results of micro-expressions are output.

[0013] Furthermore, the method also includes: performing mouth movement moment detection on the acquired video based on a speech recognition method; if the subject is detected speaking, selecting the upper face of the facial area of ​​the subject at the time of speaking to perform facial muscle movement analysis; otherwise, performing facial muscle movement analysis on the entire face of the subject.

[0014] Furthermore, the method also includes: after detecting the subject's mouth movement moment, further determining the facial area to be analyzed based on the subject's head posture; if the subject is not speaking and the subject's head deflection does not exceed a set threshold, then selecting the subject's full face for facial muscle movement analysis; otherwise, selecting the subject's half face that can be captured by the face detection algorithm for facial muscle movement analysis.

[0015] Preferably, the method for detecting the mouth movement moment of the acquired video based on the speech recognition method is as follows:

[0016] Extract text information from the original audio waveform from the audio track of the video based on the pre-trained large model;

[0017] The extracted text is time-aligned with the dialogue in the video using a phoneme alignment method;

[0018] Based on the alignment results, segment the dialogue segments in the video;

[0019] Through voiceprint feature registration and matching, the experimenter and the subject are distinguished, and the video clips of the speaker are selected;

[0020] After determining the moment when the subject speaks, the lower facial features are extracted through facial preprocessing, and the feature differences are compared based on the sliding window;

[0021] Based on the pattern matching strategy, the segments of lip shape changes are extracted to obtain the mouth movement moments.

[0022] Preferably, the facial area to be analyzed is determined based on the subject's head posture, and the specific detection method is:

[0023] A facial key point detection model is used to identify and track facial key points in real time. The deflection direction and angle of the head are determined by comparing the position changes of facial key points in consecutive frames. According to the set head rotation angle threshold, it is determined whether the deflection angle of the facial key points exceeds the angle threshold. If the deflection angle of the facial key points exceeds the angle threshold, the facial region segmentation algorithm is used to dynamically select the remaining visible side of the face in combination with the head deflection direction.

[0024] Preferably, if it is detected that the subject's head is deflected to the left or right, the left half of the face or the right half of the face is identified and separated, and the left half of the face or the right half of the face is used for evaluation according to the direction of the head deflection; if it is detected that the subject's head is deflected up or down, the upper half of the face or the lower half of the face is identified and separated, and the upper half of the face or the lower half of the face is used for evaluation according to the direction of the head deflection; based on the area of ​​interest of half the face, the GPU is used to calculate the motion field features for micro-expression detection.

[0025] Preferably, the video is segmented to obtain facial region images of each frame and facial feature point detection is performed by:

[0026] The input video is cropped into short video clips with stable head movements by using the head motion segmentation method;

[0027] After detecting the facial region of the first frame, the facial region of subsequent images is cropped based on the position of the frame;

[0028] All images are proportionally resized to the same image according to the live video stream resolution;

[0029] Perform facial feature point detection to obtain a number of facial feature points that can depict facial contours and facial features.

[0030] Preferably, the specific method of using a GPU to extract motion field features from the converted image data is:

[0031] Convert all selected interest area image data into GPU computing format data;

[0032] Using the parallel processing capability of the GPU, the motion field features corresponding to the region of interest are calculated;

[0033] Perform head motion correction on the motion field feature values ​​of all regions of interest;

[0034] The most representative motion field representation value in each region of interest is calculated to form an N×M feature matrix.

[0035] Preferably, the specific method for determining the key interval for micro-expression detection is:

[0036] Filtering is used to remove high-frequency noise and smooth the characteristic curve of each region of interest;

[0037] The upper and lower thresholds are set to screen the facial motion amplitude of each region of interest, and the candidate intervals that meet the micro-expression motion characteristics are obtained;

[0038] By determining the change pattern of the motion field characteristic curve in a window set based on the duration of micro-expression movement, the detection interval is rationally expanded or trimmed to obtain the micro-expression candidate interval that conforms to the law of movement change;

[0039] For all candidate intervals of the region of interest, the non-maximum suppression (NMS) method is used to eliminate redundant overlapping detection boxes and obtain the key intervals.

[0040] Preferably, by analyzing the regularity of facial muscle movements, local facial interest regions corresponding to the frontalis, corrugator supercilii, orbicularis oculi, levator nasolabialis, zygomaticus major, orbicularis oris and depressor anguli oris are selected for data input.

[0041] The technical solution of the present invention has the following advantages:

[0042] A. The present invention introduces a GPU-based motion field feature calculation framework in the feature calculation stage to improve the data processing speed and real-time performance of micro-expression detection. The parallel processing capability of the GPU is utilized to effectively improve the performance of micro-expression feature calculation. Through real-time processing, real-time detection of micro-expressions based on motion field features is achieved.

[0043] B. The present invention integrates speech recognition technology in the process of micro-expression detection, which can accurately identify the mouth movements of the subject during the speech process, and further improve the accuracy of micro-expression detection by eliminating data interference at these moments; after determining the speaking time of the speaker, the face is pre-processed, the lower facial features are extracted and the feature differences are compared based on the sliding window, and the lip shape changes are extracted based on a specific strategy. In the subsequent micro-expression detection process, the segments with lip shape changes will be specially processed, that is, only the upper face is focused on, to eliminate the interference of the mouth movements of the subject during the speech on the micro-expression detection, accurately locate the individual mouth movement moment, and use the upper facial features to perform micro-expression detection at such moments.

[0044] C. In a real interactive scene, the present invention captures the deflection angle and direction of the subject's head. When the head deflection angle exceeds the set angle, the present invention further captures the available half-face area for feature extraction and micro-expression recognition. At the same time, the speech recognition method is combined to eliminate the lower face area when the subject speaks, and the corresponding key facial area is selected in combination with the facial muscle movement rules to more accurately obtain the local facial area, thereby avoiding the occurrence of micro-expressions that may be effectively captured by the algorithm due to the input of the entire face or inappropriate facial area.

[0045] D. Through the application of these innovative technologies, the present invention achieves efficient and accurate micro-expression detection in various complex scenarios, providing a powerful tool for research and application in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific embodiments of the present invention, the drawings required for use in the specific embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 is a block diagram of the micro-expression detection method provided by the present invention;

[0048] Figure 2 It is a specific flow chart of determining key facial areas by combining speech recognition and head movement deflection detection provided by the present invention;

[0049] Figure 3 This is a specific flow chart of the present invention for detecting micro-expressions of the subject's mouth movements when speaking;

[0050] Figure 4 It is a schematic diagram of facial feature points for analyzing the movement rules of facial muscles provided by the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] like Figure 1 As shown, the present invention provides a real-time micro-expression detection method, and the specific method is as follows:

[0053]

S01

[0054] The camera device is connected to the computer to obtain the real-time image of the subject's head.

[0055] [S02] Segment the video, obtain facial area images of each frame, and perform facial feature point detection.

[0056] The present invention crops the input video into short video clips of set time length and stable head movement through a head movement segmentation method, such as 1 second length clips (N frames). For the convenience of calculation, after detecting the facial area of ​​the first frame, the facial area of ​​subsequent images is cropped based on the frame position. According to the real-time video stream resolution, all images are resized to 256×256 images, and of course they can also be adjusted to images of other sizes. Preferably, MediaPipe is used to detect facial feature points to obtain 468 facial feature points that can depict facial contours and facial features.

[0057]

S03

[0058] Based on the movement rules of facial muscles, M regions of interest with a size of 20×20 are selected from the eyebrows, nose, cheeks, mouth and jaw.

[0059] like Figure 4 As shown, based on a large number of facial feature points that depict facial contours and facial features, the present invention analyzes the laws of facial muscle movements, selects facial local interest regions corresponding to the frontalis, corrugator supercilii, orbicularis oculi, levator nasolabialis, zygomaticus major, orbicularis oris and depressor anguli oris, inputs data, and then performs subsequent motion field feature calculations, which can effectively avoid interference from the entire face or irrelevant areas.

[0060]

S04

[0061]

S05

[0062]

S051

[0063] [S052] Perform head motion correction on the motion field characteristic values ​​of all interest regions; the present invention preferably performs head motion correction on other interest regions based on the motion field characteristic values ​​of the nose region.

[0064] [S053] Calculate the most representative motion field representation value in each region of interest to form a feature matrix. For example, preferably select the first 20% of the optical flow in each region of interest to calculate the modulus of the horizontal optical flow and the vertical optical flow, obtain the motion field representation value of each region of interest in each frame, and form an N×M feature matrix.

[0065] [S06] The extracted feature values ​​are subjected to multi-layer filtering, interval screening and feature fusion to determine the key interval for micro-expression detection. The specific method is:

[0066]

S061

[0067] [S062] An upper threshold and a lower threshold are set to filter the facial movement amplitude of each region of interest, on the one hand, to remove slight head movements, and on the other hand, to remove extreme expressions with large movements, so as to obtain candidate intervals that meet the characteristics of micro-expression movement.

[0068] [S063] By determining the change pattern of the motion field characteristic curve in a time window set based on the duration of micro-expression movement, such as performing peak and trough and slope detection, the detection interval is rationally expanded or trimmed to obtain a micro-expression candidate interval that conforms to the law of motion change, that is, the motion amplitude changes from small to large and then to small.

[0069] [S064] For all candidate intervals of the region of interest, the non-maximum suppression (NMS) method is used to eliminate redundant overlapping detection frames, improve the accuracy of detection, and obtain the key intervals.

[0070]

S07

[0071] In the feature calculation stage, the present invention introduces a GPU-based motion field feature calculation framework, which is designed specifically to improve the data processing speed and real-time performance of micro-expression detection. First, the captured facial area image data is converted into a GPU calculation format to quickly process a large number of continuous images. Then, the parallel processing capability of the GPU is used to efficiently extract motion field features from the converted image data. This technology not only speeds up the feature calculation speed, but also realizes instant detection of micro-expressions through real-time processing. Subsequently, the most representative feature values ​​are screened out from the extracted motion field features through a feature selection algorithm. These feature values ​​are processed through multi-layer filtering, interval screening and feature fusion to finally determine the key intervals for micro-expression detection.

[0072] As a further preferred embodiment of the present invention, the present invention further integrates speech recognition technology, which can accurately identify mouth movements during speaking, and further improves the accuracy of micro-expression detection by eliminating data interference at these moments. The present invention detects the mouth movements of the acquired video based on the speech recognition method. If the subject is detected speaking, the upper face of the face area of ​​the subject at the time of speaking is selected for facial muscle movement analysis. Otherwise, the facial muscle movement analysis is performed on the entire face of the subject. The specific method is combined with Figure 3 The description is as follows:

[0073] The present invention is based on a multi-channel algorithm of text, speech and visual signals, which can capture the changes in lip shape and detect the micro-expressions of the upper face during speaking. The specific method is:

[0074] First, the Wav2Vec network is used to extract text information from the original audio waveform from the audio track of the video;

[0075] The extracted text is time-aligned with the dialogue in the video using phoneme alignment techniques.

[0076] Then, based on the alignment results, the dialogue segments in the video are segmented. On the one hand, since micro-expressions are often generated during the interaction, the emotional clues reflected in the video are learned by extracting features from the video and voice signals at this stage. On the other hand, the subject (questioner) and the subject (micro-expression research subject) are distinguished through voiceprint feature registration and matching, and the video segments when the video is speaking are selected;

[0077] After determining the speaker's speaking moment, facial features are extracted through facial preprocessing and feature differences are compared based on a sliding window, and lip shape changes are extracted based on a specific strategy;

[0078] In the subsequent micro-expression detection process, the segments with lip shape changes will be specially processed, that is, only focusing on the upper face.

[0079] As a further preferred embodiment of the present invention, the present invention further provides a method for detecting micro-expressions with large head movements. When faced with inevitable head movements in real scenes, the present invention uses advanced image processing and machine learning techniques to intelligently identify and accurately analyze the specific movement direction of the head. Specifically, it includes: after detecting the subject's mouth movements at all times, further determining the facial area to be analyzed based on the subject's head posture; if the subject is not speaking and the subject's head deflection does not exceed the set threshold, the subject's full face is selected for facial muscle movement analysis; otherwise, the half face of the subject that can be captured by the face detection algorithm is selected for facial muscle movement analysis.

[0080] Among them: the facial area to be analyzed is determined according to the subject's head posture, and the specific detection method is:

[0081] A facial key point detection model is used to identify and track facial key points in real time; the direction and angle of head deflection are determined by comparing the position changes of facial key points in consecutive frames; based on the set head rotation angle threshold (such as 45°), it is determined whether the deflection angle of facial key points exceeds the angle threshold (such as whether the relative movement of the corners of the eyes and the ears exceeds this angle threshold); if the deflection angle of facial key points exceeds the angle threshold, it is determined to be a large deflection. Once it is determined that the head has a large deflection, the facial region segmentation algorithm is used to dynamically select the remaining visible side of the face in combination with the head deflection direction.

[0082] If the subject's head is detected to be tilted left or right, the left or right half of the face is identified and separated, and the left or right half of the face is used for evaluation based on the direction of the head tilt; if the subject's head is detected to be tilted up or down, the upper or lower half of the face is identified and separated, and the upper or lower half of the face is used for evaluation based on the direction of the head tilt; after determining the facial area to be analyzed, based on the area of ​​interest of half the face, the GPU is used to calculate the motion field features and micro-expression detection is achieved through technical means such as filtering.

[0083] In summary, the present invention first analyzes the laws of facial muscle movements and accurately selects key facial areas for data input, effectively avoiding interference from the entire face or irrelevant areas. In the feature calculation stage, the present invention introduces an advanced GPU-based motion field feature calculation framework, which is designed to improve the data processing speed and real-time performance of micro-expression detection. It not only speeds up the feature calculation speed, but also realizes instant detection of micro-expressions through real-time processing.

[0084] When faced with the inevitable head movement in real scenes, the present invention can intelligently identify and analyze the specific movement direction of the head. When the head deflects to the left or right by a large amplitude, it will focus on capturing and analyzing the image data of the remaining visible side of the face (left or right). Similarly, if the deflection of the head is mainly up and down, the visible upper half of the face or lower half of the face will be selected for analysis. This dynamic selection mechanism ensures that even in the case of large head movement, key micro-expression features can still be effectively extracted from the available facial area. In addition, the present invention uses image processing technology to quickly adjust and optimize feature extraction algorithms for half-face or partial face areas, ensuring that these algorithms are equally effective on non-standard facial images. This strategy significantly enhances the applicability and accuracy of the present invention in various complex situations.

[0085] At the same time, the method of the present invention also integrates speech recognition technology, which can accurately identify mouth movements during speaking. By eliminating data interference at these times, the accuracy of micro-expression detection is further improved. After determining the speaker's speaking time, facial preprocessing is used to extract lower facial features and compare feature differences based on a sliding window, and lip shape changes are extracted based on a specific strategy. In the subsequent micro-expression detection process, segments with lip shape changes will be specially processed, that is, only the upper face will be focused on. Through the application of innovative technologies, efficient and accurate micro-expression detection in various complex scenarios is achieved, providing a powerful tool for research and application in related fields.

[0086] The matters not described in the present invention are applicable to the prior art.

[0087] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. However, the obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A real-time micro-expression detection method, characterized in that: The method comprises: Get real-time video; Segment the video, obtain the facial region image of each frame, and perform facial feature point detection; Analyze the movement patterns of facial muscles and select key facial areas for data input; Convert the captured facial area image data into GPU computing format; Use GPU to extract motion field features from the converted image data and select the most representative feature values; The extracted feature values ​​are subjected to multi-layer filtering, interval screening and feature fusion to determine the key interval for micro-expression detection; Combined with the key intervals of micro-expression detection, the real-time detection results of micro-expressions are output.

2. The micro-expression real-time detection method according to claim 1, characterized in that: The method also includes: performing mouth movement moment detection on the acquired video based on a speech recognition method, and if it is detected that the subject is speaking, selecting the upper face of the facial area of ​​the subject at the time of speaking to perform facial muscle movement analysis, otherwise, performing facial muscle movement analysis on the entire face of the subject.

3. The micro-expression real-time detection method according to claim 2, characterized in that: The method also includes: after detecting the subject's mouth movement moment, further determining the facial area to be analyzed based on the subject's head posture; if the subject is not speaking and the subject's head deflection does not exceed a set threshold, selecting the subject's full face for facial muscle movement analysis; otherwise, selecting the subject's half face that can be captured by the face detection algorithm for facial muscle movement analysis.

4. The real-time micro-expression detection method according to claim 3, characterized in that: The specific method of detecting the mouth movement moment of the acquired video based on the speech recognition method is: Extract text information from the original audio waveform from the audio track of the video based on the pre-trained large model; The extracted text is time-aligned with the dialogue in the video using a phoneme alignment method; Based on the alignment results, segment the dialogue segments in the video; Through voiceprint feature registration and matching, the experimenter and the subject are distinguished, and the video clips of the speaker are selected; After determining the moment when the subject speaks, the lower facial features are extracted through facial preprocessing, and the feature differences are compared based on the sliding window; Based on the pattern matching strategy, the segments of lip shape changes are extracted to obtain the mouth movement moments.

5. The micro-expression real-time detection method according to claim 3 or 4, characterized in that: The specific detection method for determining the facial area to be analyzed based on the subject's head posture is: Use the facial key point detection model to identify and track facial key points in real time; determine the deflection direction and angle of the head by comparing the position changes of facial key points in consecutive frames; determine whether the deflection angle of facial key points exceeds the angle threshold according to the set head rotation angle threshold; If the deflection angle of the facial key points exceeds the angle threshold, the facial region segmentation algorithm is used to dynamically select the remaining visible side of the face in combination with the head deflection direction.

6. The real-time micro-expression detection method according to claim 5, characterized in that: If the subject's head is detected to be tilted left or right, the left or right half of the face is identified and separated, and the left or right half of the face is used for evaluation based on the direction of the head tilt; if the subject's head is detected to be tilted up or down, the upper or lower half of the face is identified and separated, and the upper or lower half of the face is used for evaluation based on the direction of the head tilt; based on the area of ​​interest of half the face, the GPU is used to calculate the motion field features for micro-expression detection.

7. The real-time micro-expression detection method according to claim 1, characterized in that: The specific method of segmenting the video, obtaining facial region images of each frame, and detecting facial feature points is as follows: The input video is cropped into short video clips of a set time length and with stable head movements by using the head motion segmentation method; After detecting the facial region of the first frame, the facial region of subsequent images is cropped based on the position of the frame; Resize all images proportionally to the same size according to the live video stream resolution; Perform facial feature point detection to obtain a number of facial feature points that can depict facial contours and facial features.

8. The micro-expression real-time detection method according to claim 1, characterized in that: The specific method of using GPU to extract motion field features from the converted image data is as follows: Convert all selected interest area image data into GPU computing format data; Using the parallel processing capability of the GPU, the motion field features corresponding to the region of interest are calculated; Perform head motion correction on the motion field feature values ​​of all regions of interest; The most representative motion field representation value in each region of interest is calculated to form an N×M feature matrix.

9. The real-time micro-expression detection method according to claim 1, characterized in that: The specific method for determining the key interval for micro-expression detection is: Filtering is used to remove high-frequency noise and smooth the characteristic curve of each region of interest; The upper and lower thresholds are set to screen the facial motion amplitude of each region of interest, and the candidate intervals that meet the micro-expression motion characteristics are obtained; By determining the change pattern of the motion field characteristic curve in a window set based on the duration of micro-expression movement, the detection interval is rationally expanded or trimmed to obtain the micro-expression candidate interval that conforms to the law of movement change; For all candidate intervals of the region of interest, the non-maximum suppression (NMS) method is used to eliminate redundant overlapping detection boxes and obtain the key intervals.

10. The real-time micro-expression detection method according to claim 1, characterized in that: By analyzing the laws of facial muscle movements, local facial interest regions corresponding to the frontalis, corrugator supercilii, orbicularis oculi, levator nasolabialis, zygomaticus major, orbicularis oris and depressor anguli oris were selected for data input.

Images