An expression detection method, system, device and storage medium

Abstract

The application discloses an expression detection method, system, device and storage medium, the expression detection method comprises the feature point detection to the first reference frame, calculates the first optical flow intensity value of each video frame of the preset subregion according to the first reference frame;According to the first optical flow intensity value and the preset stage threshold value, the video frame interval screening is carried out, and the first video frame candidate interval is obtained;The second optical flow intensity value of the preset subregion in each video frame in the second video frame candidate interval is calculated according to the second reference frame, and the video frame interval screening is carried out according to the second optical flow intensity value and the preset stage threshold value, and the third video frame candidate interval is obtained;According to the third video frame candidate interval and the preset reconstruction threshold value, the video frame interval screening is carried out through the empirical mode decomposition algorithm, and the micro-expression detection interval is obtained, the micro-expression detection interval is used as the expression detection result of the video to be detected, the misjudgment quantity of the detection result is reduced, and the accuracy of detection is improved.

Description

Technical Field

[0001] This invention relates to the field of facial expression detection technology, and in particular to a facial expression detection method, system, device, and storage medium. Background Technology

[0002] The expression detection task is to locate the intervals in which macro-expressions (MaE) and micro-expressions (ME) occur in long videos.

[0003] Currently, facial expression detection technology is mainly divided into two categories: deep learning methods and traditional methods. The most outstanding methods in both categories use optical flow to capture facial motion features. However, micro-expression datasets are relatively small, and the annotations differ between different datasets. These issues make training a highly generalizable deep learning model extremely difficult. Therefore, in recent facial expression detection competitions, the best-performing methods are still the traditional methods based on optical flow intensity features. The best-performing traditional methods use a sliding window to process long videos, converting the optical flow extracted from the feature regions into intensity signals, then using low-pass filtering to denoise the signals, and finally detecting expressions by analyzing the intensity signals.

[0004] However, the sliding window scheme uses the first frame within the window as a reference (reference frame) to calculate optical flow for the remaining frames. This means it captures the differences between each frame within the window and the first frame through optical flow, and determines whether there is facial expression in the current window by observing changes in the optical flow intensity signal. Existing schemes typically crop out the face region of each frame and use alignment schemes to keep the cropped face as still as possible (removing the influence of head movement). However, the most effective alignment schemes currently only effectively address the 2D translation problem of the head. The influence of head rotation remains. Therefore, even without facial expression, frames far from the reference frame within the window may accumulate significant differences, leading to a continuous increase in the extracted optical flow intensity within the window. This makes traditional methods for detecting facial expressions based on changes in optical flow intensity very prone to false positives; moreover, after facial expressions cause deformation (such as frowning), there will be a large difference between the current frame and the reference frame, resulting in a higher calculated optical flow intensity between the two frames. However, facial deformation after an expression may not be fully restored, resulting in a significant difference from the reference frame. In this case, even minor changes in subsequent frames may lead to very high calculated optical flow intensities, which can easily cause false detections. Furthermore, slow head movements can also increase the difference between the current frame and the base frame, thereby increasing the optical flow intensity within the window. Since the current optimal solution only uses changes in optical flow intensity to determine whether an expression has occurred, the increase in optical flow intensity caused by slow head movements can also easily lead to false detections. Summary of the Invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art. To this end, the present invention proposes an expression detection method, system, device, and storage medium, which can reduce the number of false detections and improve the accuracy of detection.

[0006] A first aspect of the present invention provides an expression detection method, comprising the following steps:

[0007] Obtain the video to be tested;

[0008] A sliding window is established, and the starting point of the sliding window is set at the position of the starting frame of the video to be detected, wherein the sliding window covers a preset number of video frames;

[0009] Feature point detection is performed on the first reference frame to obtain several preset sub-regions within the face region, wherein the first reference frame is the first video frame in the sliding window, and the first subsequent frames are all video frames in the sliding window except for the reference frame.

[0010] Calculate the first optical flow intensity value of a preset sub-region of each video frame based on the first reference frame;

[0011] Based on the first optical flow intensity value and the preset stage threshold, the video frame interval is filtered to obtain the first video frame candidate interval;

[0012] The second optical flow intensity value of a preset sub-region of each video frame within the second video frame candidate interval is calculated based on the second reference frame. The second video frame candidate interval is composed of the first video frame candidate interval and video frames extended to a preset number of future times. The second reference frame is the first video frame of the first video frame candidate interval.

[0013] Based on the second optical flow intensity value and the preset stage threshold, video frame intervals are filtered to obtain a third video frame candidate interval;

[0014] Based on the candidate interval of the third video frame and the preset reconstruction threshold, the video frame interval is filtered by the empirical mode decomposition algorithm to obtain the micro-expression detection interval, and the micro-expression detection interval is used as the expression detection result of the video to be detected.

[0015] The control method according to embodiments of the present invention has at least the following beneficial effects:

[0016] This method first establishes a sliding window, setting its starting point at the beginning frame of the video to be detected. The sliding window covers a preset number of video frames. Feature point detection is performed on a first reference frame to obtain several preset sub-regions within the face area. The first reference frame is the first video frame in the sliding window, and the first subsequent frames are all video frames in the sliding window except the reference frame. A first optical flow intensity value is calculated for each preset sub-region of each video frame based on the first reference frame. Video frame intervals are filtered based on the first optical flow intensity value and a preset stage threshold to obtain a first video frame candidate interval. A second optical flow intensity value is calculated for each preset sub-region of each video frame within the second video frame candidate interval based on a second reference frame. The second video frame candidate interval consists of the first video frame candidate interval and a preset number of video frames extended to future times. The second reference frame is the first video frame of the first video frame candidate interval. The second optical flow intensity value is then calculated based on the second optical flow intensity value and a preset stage threshold. The application uses a two-stage optical flow strategy. First, a sliding window scheme is used to select intervals with higher optical flow intensity as candidate intervals. A separate reference frame is set for each candidate interval. Then, face alignment is performed again based on the new reference frame to reduce the influence of actions unrelated to facial expressions. At the same time, the window size for the second-stage optical flow extraction is dynamically adjusted so that the window range just covers the candidate interval, thereby improving the problem of high optical flow intensity caused by error accumulation. Then, based on the candidate intervals of the third video frame and the preset reconstruction threshold, the video frame intervals are selected using the empirical mode decomposition algorithm to obtain the micro-expression detection interval. The spectral analysis method is used to identify whether the interval contains facial expressions, which can better identify facial expressions and slow head movements. The micro-expression detection interval is used as the expression detection result of the video to be detected, reducing the number of false detections and improving the detection accuracy.

[0017] According to some embodiments of the present invention, the preset sub-region includes a nose tip sub-region and other sub-regions besides the nose tip sub-region, and the step of calculating the first optical flow intensity value of the preset sub-region of each video frame based on the first reference frame includes:

[0018] Calculate the optical flow intensity value of the nose tip region between each of the first subsequent frames and the first reference frame;

[0019] The first subsequent frame is aligned with the first reference frame based on the optical flow intensity value of the nose tip region to obtain the first aligned video frame;

[0020] Calculate the optical flow intensity values ​​of the other sub-regions between the first aligned video frame and the first reference frame;

[0021] Calculate the average value of the optical flow intensity value for each of the other sub-regions, and use the average value as the region optical flow intensity value corresponding to each of the other sub-regions;

[0022] High-frequency noise in the optical flow intensity value corresponding to each of the other sub-regions is removed by low-pass filtering to obtain the first optical flow intensity value of the preset sub-region of each video frame.

[0023] According to some embodiments of the present invention, the preset stage threshold includes preset stage threshold one, preset stage threshold two, and preset stage threshold three. The step of filtering video frame intervals based on the first optical flow intensity value and the preset stage thresholds to obtain a first video frame candidate interval includes:

[0024] Find the first video frame interval where the first optical flow intensity value is greater than the preset stage threshold and the number of video frames exceeds the preset number of video frames;

[0025] Find the first peak value in the first video frame interval;

[0026] Subtract the preset stage threshold from the first wave peak value to obtain the first difference;

[0027] Find the second video frame interval where the first difference is greater than the preset stage threshold two;

[0028] Find the second peak value in the second video frame interval, and take the video frame where the second peak value is located as the first starting frame;

[0029] From the first starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the preset video frame number range towards the historical time, and take it as the first starting position of the second video frame range.

[0030] From the first starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the range of the preset number of video frames, and take it as the first end position of the second video frame range.

[0031] The first video frame candidate range is obtained based on the first start position and the first end position.

[0032] According to some embodiments of the present invention, the step of filtering video frame intervals based on the second optical flow intensity value and the preset stage threshold to obtain a third video frame candidate interval includes:

[0033] Find a third video frame interval where the second optical flow intensity value is greater than the preset stage threshold and the number of video frames exceeds the preset number of video frames;

[0034] Find the third peak value in the third video frame interval;

[0035] Subtract the preset stage threshold from the third wave peak value to obtain the second difference;

[0036] Find the fourth video frame interval where the second difference is greater than the preset stage threshold two;

[0037] Find the fourth peak value in the fourth video frame interval, and take the video frame where the fourth peak value is located as the second starting frame;

[0038] From the second starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the preset video frame number range, and take it as the second starting position of the second video frame range.

[0039] From the first starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the range of the preset number of video frames, and take it as the second end position of the second video frame range.

[0040] The third video frame candidate range is obtained based on the second start position and the second end position.

[0041] According to some embodiments of the present invention, the preset sub-region includes a nose tip sub-region and other sub-regions besides the nose tip sub-region. The step of calculating the second optical flow intensity value of the preset sub-region for each video frame within the second video frame candidate interval based on the second reference frame, wherein the second video frame candidate interval is composed of the first video frame candidate interval and video frames extended a preset number of times into the future, and the second reference frame is the first video frame of the first video frame candidate interval, including:

[0042] Calculate the optical flow intensity value of the nose tip region between each video frame in the second video frame candidate interval and the second reference frame;

[0043] Based on the optical flow intensity value of the nose tip region, each video frame in the second video frame candidate interval is aligned with the second reference frame to obtain the second aligned video frame;

[0044] Calculate the optical flow intensity values ​​of the other sub-regions between the second aligned video frame and the second reference frame;

[0045] Calculate the average value of the optical flow intensity value for each of the other sub-regions, and use the average value as the region optical flow intensity value corresponding to each of the other sub-regions;

[0046] High-frequency noise in the optical flow intensity value corresponding to each of the other sub-regions is removed by low-pass filtering to obtain the second optical flow intensity value of the preset sub-region of each video frame.

[0047] According to some embodiments of the present invention, the step of filtering video frame intervals based on the third video frame candidate interval and a preset reconstruction threshold using an empirical mode decomposition algorithm to obtain micro-expression detection intervals includes:

[0048] The second optical flow intensity value of the candidate interval of the third video frame is decomposed by empirical mode decomposition algorithm to obtain the decomposed intrinsic mode function of the candidate interval of the third video frame.

[0049] The lowest frequency intrinsic mode function of the decomposed intrinsic mode function is removed to obtain the removed intrinsic mode function of the candidate interval of the third video frame;

[0050] The removed intrinsic mode functions are reconstructed to obtain the reconstructed signal of the candidate interval of the third video frame;

[0051] The micro-expression detection interval is obtained by filtering the video frame intervals whose reconstructed signals are greater than the preset reconstruction threshold.

[0052] According to some embodiments of the present invention, the expression detection method further includes:

[0053] When the preset number of video frames is less than the number of video frames in the video to be tested, the sliding window is moved along the time axis of the video to be tested with a preset step size. Each time the sliding window is moved, the micro-expression detection interval of the current sliding window is calculated based on the first video frame of the current sliding window. This process is repeated until the starting point of the sliding window is outside the video to be tested. The micro-expression detection intervals of all sliding windows are then used as the expression detection result of the video to be tested.

[0054] A second aspect of the present invention provides an expression detection system, the expression detection system comprising:

[0055] The data acquisition module is used to acquire the video to be detected;

[0056] A sliding window creation module is used to create a sliding window and set the starting point of the sliding window at the position of the starting frame of the video to be detected, wherein the sliding window covers a preset number of video frames;

[0057] The feature point detection module is used to perform feature point detection on the first reference frame to obtain several preset sub-regions within the face region, wherein the first reference frame is the first video frame in the sliding window, and the first subsequent frames are all video frames in the sliding window except for the reference frame.

[0058] The first optical flow intensity value calculation module is used to calculate the first optical flow intensity value of a preset sub-region of each video frame based on the first reference frame;

[0059] The first video frame candidate interval filtering module is used to filter video frame intervals based on the first optical flow intensity value and a preset stage threshold to obtain the first video frame candidate interval.

[0060] The second optical flow intensity value calculation module is used to calculate the second optical flow intensity value of a preset sub-region of each video frame in the second video frame candidate interval according to the second reference frame. The second video frame candidate interval is composed of the first video frame candidate interval and video frames extended to a preset number of future times. The second reference frame is the first video frame of the first video frame candidate interval.

[0061] The third video frame candidate interval filtering module is used to filter video frame intervals based on the second optical flow intensity value and the preset stage threshold to obtain the third video frame candidate interval.

[0062] The expression detection module is used to filter video frame intervals using an empirical mode decomposition algorithm based on the candidate interval of the third video frame and a preset reconstruction threshold to obtain micro-expression detection intervals, and to use the micro-expression detection intervals as the expression detection results of the video to be detected.

[0063] This system first establishes a sliding window, setting its starting point at the beginning frame of the video to be detected. The sliding window covers a preset number of video frames. Feature point detection is performed on a first reference frame to obtain several preset sub-regions within the face area. The first reference frame is the first video frame in the sliding window, and the first subsequent frames are all video frames in the sliding window except the reference frame. A first optical flow intensity value is calculated for each preset sub-region of each video frame based on the first reference frame. Video frame intervals are filtered based on the first optical flow intensity value and a preset stage threshold to obtain a first video frame candidate interval. A second optical flow intensity value is calculated for each preset sub-region of a video frame within the second video frame candidate interval based on a second reference frame. The second video frame candidate interval consists of the first video frame candidate interval and a preset number of video frames extended to future times. The second reference frame is the first video frame of the first video frame candidate interval. The system then calculates the second optical flow intensity value based on the second optical flow intensity value and a preset stage threshold. The application uses a two-stage optical flow strategy. First, a sliding window scheme is used to select intervals with higher optical flow intensity as candidate intervals. A separate reference frame is set for each candidate interval. Then, face alignment is performed again based on the new reference frame to reduce the influence of actions unrelated to facial expressions. At the same time, the window size for the second-stage optical flow extraction is dynamically adjusted so that the window range just covers the candidate interval, thereby improving the problem of high optical flow intensity caused by error accumulation. Then, based on the candidate intervals of the third video frame and the preset reconstruction threshold, the video frame intervals are selected using the empirical mode decomposition algorithm to obtain the micro-expression detection interval. The spectral analysis method is used to identify whether the interval contains facial expressions, which can better identify facial expressions and slow head movements. The micro-expression detection interval is used as the expression detection result of the video to be detected, reducing the number of false detections and improving the detection accuracy.

[0064] A third aspect of the present invention provides an electronic expression detection device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the expression detection method described above.

[0065] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-described facial expression detection method.

[0066] It should be noted that the beneficial effects of the second to fourth aspects of the present invention compared with the prior art are the same as the beneficial effects of the above-described expression detection system compared with the prior art, and will not be described in detail here.

[0067] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0068] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0069] Figure 1 This is a flowchart of an expression detection method according to an embodiment of the present invention;

[0070] Figure 2 This is a schematic diagram of facial feature points and feature regions in an expression detection method provided by an embodiment of the present invention;

[0071] Figure 3 This is a schematic diagram of the candidate interval selection for an expression detection method provided in an embodiment of the present invention;

[0072] Figure 4 This is a diagram showing the reconstructed signal filtering and detection results of an expression detection method provided in this embodiment of the invention.

[0073] Figure 5 This is a schematic diagram of the first optical flow signal intensity extracted by the sliding window and the second optical flow intensity value extracted in the candidate interval of the second video frame in an expression detection method provided by an embodiment of the present invention.

[0074] Figure 6 This is a schematic diagram of the reconstructed signal of a real facial expression from an embodiment of the facial expression detection method provided by the present invention;

[0075] Figure 7 This is a schematic diagram of the reconstructed signal of slow head movement in an expression detection method provided in an embodiment of the present invention;

[0076] Figure 8 This is a schematic diagram of the structure of an expression detection system according to an embodiment of the present invention. Detailed Implementation

[0077] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0078] In the description of this invention, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated.

[0079] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0080] In the description of this invention, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0081] Before introducing this application, let's first introduce the sliding window, optical flow, low-pass filtering, and EMD methods:

[0082] Sliding window: Since there are many frames in a video, it is difficult to process all of them at once. Therefore, a relatively small window is selected, which processes a specific number of frames in the video each time (e.g., 200 frames). After processing, the same number of subsequent frames are selected for processing. The whole process is somewhat similar to a fixed window sliding along the time dimension of the video.

[0083] Optical flow: In computer vision, it defines the movement of an object in an image, which can be caused by camera movement or object movement. Specifically, it refers to the amount of movement of a pixel representing the same object in one frame of a video image to the next frame.

[0084] Low-pass filtering: Used to remove high-frequency noise from a signal.

[0085] EMD (Empirical Mode Decomposition) is an algorithm well-suited for processing nonlinear, non-stationary time series. Essentially, it's a method to stationarize data sequences or signals. The goal of the EMD algorithm is to decompose a signal into multiple components called "Intrinsic Mode Functions" (IMFs), each representing a different frequency component of the signal.

[0086] Currently, facial expression detection technology is mainly divided into two categories: deep learning methods and traditional methods. The most outstanding methods in both categories use optical flow to capture facial motion features. However, micro-expression datasets are relatively small, and the annotations differ between different datasets. These issues make training a highly generalizable deep learning model extremely difficult. Therefore, in recent facial expression detection competitions, the best-performing methods are still the traditional methods based on optical flow intensity features. The best-performing traditional methods use a sliding window to process long videos, converting the optical flow extracted from the feature regions into intensity signals, then using low-pass filtering to denoise the signals, and finally detecting expressions by analyzing the intensity signals.

[0087] However, the sliding window scheme uses the first frame within the window as a reference (reference frame) to calculate optical flow for the remaining frames. This means it captures the differences between each frame within the window and the first frame through optical flow, and determines whether there is facial expression in the current window by observing changes in the optical flow intensity signal. Existing schemes typically crop out the face region of each frame and use alignment schemes to keep the cropped face as still as possible (removing the influence of head movement). However, the most effective alignment schemes currently only effectively address the 2D translation problem of the head. The influence of head rotation remains. Therefore, even without facial expression, frames far from the reference frame within the window may accumulate significant differences, leading to a continuous increase in the extracted optical flow intensity within the window. This makes traditional methods for detecting facial expressions based on changes in optical flow intensity very prone to false positives; moreover, after facial expressions cause deformation (such as frowning), there will be a large difference between the current frame and the reference frame, resulting in a higher calculated optical flow intensity between the two frames. However, facial deformation after an expression may not be fully restored, resulting in a significant difference from the reference frame. In this case, even minor changes in subsequent frames may lead to very high calculated optical flow intensities, which can easily cause false detections. Furthermore, slow head movements can also increase the difference between the current frame and the base frame, thereby increasing the optical flow intensity within the window. Since the current optimal solution only uses changes in optical flow intensity to determine whether an expression has occurred, the increase in optical flow intensity caused by slow head movements can also easily lead to false detections.

[0088] To address the aforementioned technical deficiencies, referring to... Figure 1 This invention provides an expression detection method, comprising:

[0089] Step S101: Obtain the video to be tested;

[0090] Step S102: Establish a sliding window and set the starting point of the sliding window at the position of the starting frame of the video to be detected, wherein the sliding window covers a preset number of video frames;

[0091] Step S103: Perform feature point detection on the first reference frame to obtain several preset sub-regions within the face region, wherein the first reference frame is the first video frame in the sliding window, and the first subsequent frame is all video frames in the sliding window except for the reference frame.

[0092] Step S104: Calculate the first optical flow intensity value of the preset sub-region of each video frame based on the first reference frame;

[0093] Step S105: Based on the first optical flow intensity value and the preset stage threshold, the video frame interval is filtered to obtain the first video frame candidate interval;

[0094] Step S106: Calculate the second optical flow intensity value of a preset sub-region of each video frame in the second video frame candidate interval according to the second reference frame, wherein the second video frame candidate interval is composed of the first video frame candidate interval and video frames extended to a preset number of future times, and the second reference frame is the first video frame of the first video frame candidate interval.

[0095] Step S107: Based on the second optical flow intensity value and the preset stage threshold, the video frame interval is filtered to obtain the third video frame candidate interval;

[0096] Step S108: Based on the candidate interval of the third video frame and the preset reconstruction threshold, the video frame interval is filtered by the empirical mode decomposition algorithm to obtain the micro-expression detection interval, and the micro-expression detection interval is used as the expression detection result of the video to be detected.

[0097] It should be noted that, in some embodiments, the expression detection method further includes:

[0098] When the preset number of video frames is less than the number of video frames in the video to be detected, the sliding window is moved along the time axis of the video to be detected with a preset step size. Each time the sliding window is moved, steps S102 to S108 are repeated until the starting point of the sliding window is outside the video to be detected, and the micro-expression detection range of all sliding windows is taken as the expression detection result of the video to be detected.

[0099] This method first establishes a sliding window, setting its starting point at the beginning frame of the video to be detected. The sliding window covers a preset number of video frames. Feature point detection is performed on a first reference frame to obtain several preset sub-regions within the face area. The first reference frame is the first video frame in the sliding window, and the first subsequent frames are all video frames in the sliding window except the reference frame. A first optical flow intensity value is calculated for each preset sub-region of each video frame based on the first reference frame. Video frame intervals are filtered based on the first optical flow intensity value and a preset stage threshold to obtain a first video frame candidate interval. A second optical flow intensity value is calculated for each preset sub-region of each video frame within the second video frame candidate interval based on a second reference frame. The second video frame candidate interval consists of the first video frame candidate interval and a preset number of video frames extended to future times. The second reference frame is the first video frame of the first video frame candidate interval. The second optical flow intensity value is then calculated based on the second optical flow intensity value and a preset stage threshold. The application uses a two-stage optical flow strategy. First, a sliding window scheme is used to select intervals with higher optical flow intensity as candidate intervals. A separate reference frame is set for each candidate interval. Then, face alignment is performed again based on the new reference frame to reduce the influence of actions unrelated to facial expressions. At the same time, the window size for the second-stage optical flow extraction is dynamically adjusted so that the window range just covers the candidate interval, thereby improving the problem of high optical flow intensity caused by error accumulation. Then, based on the candidate intervals of the third video frame and the preset reconstruction threshold, the video frame intervals are selected using the empirical mode decomposition algorithm to obtain the micro-expression detection interval. The spectral analysis method is used to identify whether the interval contains facial expressions, which can better identify facial expressions and slow head movements. The micro-expression detection interval is used as the expression detection result of the video to be detected, reducing the number of false detections and improving the detection accuracy.

[0100] In some embodiments, the preset sub-region includes a nose tip sub-region and other sub-regions besides the nose tip sub-region. In step S104, calculating the first optical flow intensity value of the preset sub-region for each video frame based on the first reference frame includes:

[0101] Step S201: Calculate the optical flow intensity value of the nose tip region between each first subsequent frame and the first reference frame;

[0102] Step S202: Align the first subsequent frame with the first reference frame based on the optical flow intensity value of the nose tip region to obtain the first aligned video frame;

[0103] Step S203: Calculate the optical flow intensity values ​​of other sub-regions between the first aligned video frame and the first reference frame;

[0104] Step S204: Calculate the average value of the optical flow intensity of each other sub-region, and use the average value as the region optical flow intensity value corresponding to each other sub-region;

[0105] Step S205: Remove high-frequency noise from the optical flow intensity values ​​of each other sub-region by using fast Fourier transform and low-pass filter to obtain the first optical flow intensity value of the preset sub-region of each video frame.

[0106] Specifically, refer to Figure 2 In some embodiments, a sliding window scheme is used to process long videos. For each window, the first frame is used as the reference frame. The face alignment tool is used to detect 68 facial feature points. Based on these feature points, the facial region, as well as the positions and sizes of five feature regions—two eyebrows, two sides of the nose, and the mouth—are determined. Using the reference frame as a reference, other frames are cropped and aligned, and optical flow is calculated. Then, the average intensity of the optical flow extracted from each feature region is calculated to obtain the optical flow intensity signal of each feature region. High-frequency noise in the signal is removed by low-pass filtering. The Farneback algorithm is used to calculate the dense optical flow between the feature regions of each frame and the reference frame. The average optical flow intensity of different feature regions is calculated separately, so each window can obtain five optical flow intensity curves, and the five feature regions correspond one-to-one with the five optical flow intensity curves.

[0107] This embodiment uses a sliding window scheme to filter regions with higher optical flow intensity as candidate regions for the first time, and sets a separate reference frame for the candidate regions. This reduces the influence of actions unrelated to facial expressions, thereby improving the problem of high optical flow intensity caused by error accumulation.

[0108] Reference Figure 3 In some embodiments, the preset stage thresholds include preset stage threshold one, preset stage threshold two, and preset stage threshold three. In step S105, video frame intervals are filtered based on the first optical flow intensity value and the preset stage thresholds to obtain a first video frame candidate interval, including:

[0109] Step S301: Find the first video frame interval where the first optical flow intensity value is greater than a preset stage threshold and the number of video frames exceeds a preset number of video frames.

[0110] Step S302: Find the first peak value in the first video frame interval;

[0111] Step S303: Subtract the preset stage threshold one from the first wave peak value to obtain the first difference;

[0112] Step S304: Find the second video frame interval where the first difference is greater than the preset stage threshold two;

[0113] Step S305: Find the second peak value in the second video frame interval, and take the video frame where the second peak value is located as the first starting frame;

[0114] Step S306: Find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the range of preset video frame number intervals from the first starting frame to the historical time, and take it as the first starting position of the second video frame interval.

[0115] Step S307: Find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the range of preset video frame number intervals from the first starting frame to the future time, and take it as the first end position of the second video frame interval.

[0116] Step S308: Obtain the first video frame candidate interval based on the first start position and the first end position.

[0117] In some embodiments, refer to Figure 3 , Figure 3 In the "find peak" function, "adjust intervals" is used to adjust interval boundaries, and "candidate interval" is used to select candidate intervals. Specifically, two thresholds are set to find intervals in the optical flow intensity signals of the five feature regions that exceed the threshold θ1 (preset stage threshold one). The peaks of the signals within these intervals are found. If the intensity of the peaks of the signals within these intervals minus θ1 exceeds θ2 (preset stage threshold two), then it is used as the second video frame interval; otherwise, the current interval is ignored.

[0118] For the second video frame interval, the starting position of the interval is found by looking back at the peak position towards historical time. If the optical flow intensity within frames 1-5 (within a preset number of video frames) does not decrease by more than θ3 (preset stage threshold three), then the current position (the current position is the video frame found towards historical time where the decrease does not exceed θ3) is taken as the starting position of the interval. Similarly, the ending position of the interval is found towards future time. If the optical flow intensity within frames 1-5 does not decrease by more than θ3 (preset stage threshold three), then the current position (the current position is the video frame found towards future time where the decrease does not exceed θ3) is taken as the ending position of the interval. If the intervals selected from the five feature regions have an intersection, then the intervals with the intersection are merged, and the intervals that are not merged are used together as the first video frame candidate interval.

[0119] This embodiment uses a sliding window scheme to filter intervals with higher optical flow intensity as candidate intervals, reducing the influence of actions unrelated to facial expressions. At the same time, it can serve as a reference frame for subsequent candidate intervals, thereby improving the problem of excessively high optical flow intensity caused by error accumulation.

[0120] In some embodiments, in step S107, video frame intervals are filtered based on the second optical flow intensity value and a preset stage threshold to obtain a third video frame candidate interval, including:

[0121] Step S401: Find the third video frame interval where the second optical flow intensity value is greater than the preset stage threshold and the number of video frames exceeds the preset number of video frames.

[0122] Step S402: Find the third peak value in the third video frame interval;

[0123] Step S403: Subtract the preset stage threshold one from the third wave peak value to obtain the second difference;

[0124] Step S404: Find the fourth video frame interval where the second difference is greater than the preset stage threshold two;

[0125] Step S405: Find the fourth peak value in the fourth video frame interval, and take the video frame where the fourth peak value is located as the second starting frame;

[0126] Step S406: Find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the range of preset video frame number intervals from the second starting frame to the historical time, and take it as the second starting position of the second video frame interval.

[0127] Step S407: Find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the range of preset video frame number intervals from the first starting frame to the future time, and take it as the second end position of the second video frame interval.

[0128] Step S408: Obtain the third video frame candidate interval based on the second start position and the second end position.

[0129] Specifically, the end position of each first video frame candidate interval is extended forward by 20 frames (preset number) as the window for optical flow extraction in the second stage. The first video frame of the first video frame candidate interval is used as the second reference frame. The optical flow extraction within the window is the same as in the first stage. The difference is that the second stage only targets the first video frame candidate interval. The optical flow signal in the second stage is analyzed, and intervals in the optical flow intensity signals of the five feature regions that exceed the threshold θ1 (preset stage threshold one) and have more than 10 video frames (preset video frame number) are found. The peaks of the signals in these intervals are found. If the intensity of the signal peaks in these intervals minus θ1 exceeds θ2 (preset stage threshold two), it is used as the fourth video frame interval; otherwise, the current interval is ignored.

[0130] For the fourth video frame interval, the starting position of the interval is found by looking back at the peak position. If the optical flow intensity within frames 1-5 (within the preset number of video frames) does not decrease by more than θ3 (preset stage threshold three), then the current position (the video frame found in the historical time where the decrease does not exceed θ3) is taken as the starting position of the interval. Similarly, the ending position of the interval is found looking back at future time. If the optical flow intensity within frames 1-5 does not decrease by more than θ3 (preset stage threshold three), then the current position (the video frame found looking back at the future time where the decrease does not exceed θ3) is taken as the ending position of the interval. If the intervals selected from the five feature regions have an intersection, then the intervals with the intersection are merged, and the intervals that are not merged are used together as the candidate intervals for the third video frame.

[0131] This embodiment uses a two-stage optical flow strategy. First, a sliding window scheme is used to select regions with higher optical flow intensity as candidate regions. A separate reference frame is set for each candidate region. Then, face alignment is performed again based on the new reference frame to reduce the influence of actions unrelated to facial expressions. At the same time, the window size for the second-stage optical flow extraction is dynamically adjusted so that the window range just covers the candidate regions, thereby improving the problem of high optical flow intensity caused by error accumulation.

[0132] In some embodiments, in step S106, a second optical flow intensity value of a preset sub-region of each video frame within the second video frame candidate interval is calculated based on the second reference frame. The second video frame candidate interval is composed of a first video frame candidate interval and video frames extended a preset number of times into the future. The second reference frame is the first video frame of the first video frame candidate interval, including:

[0133] Step S501: Calculate the optical flow intensity value of the nose tip region between each video frame and the second reference frame within the candidate interval of the second video frame;

[0134] Step S502: Align each video frame in the candidate interval of the second video frame with the second reference frame based on the optical flow intensity value of the nose tip region to obtain the second aligned video frame;

[0135] Step S503: Calculate the optical flow intensity values ​​of other sub-regions between the second aligned video frame and the second reference frame;

[0136] Step S504: Calculate the average value of the optical flow intensity of each other sub-region, and use the average value as the region optical flow intensity value corresponding to each other sub-region;

[0137] Step S505: Remove high-frequency noise from the optical flow intensity values ​​of each other sub-region by using fast Fourier transform and low-pass filter to obtain the second optical flow intensity value of the preset sub-region of each video frame.

[0138] This embodiment sets a separate reference frame for the candidate interval, and then performs face alignment again based on the new reference frame to reduce the influence of actions unrelated to facial expressions. At the same time, it dynamically adjusts the window size of the second-stage optical flow extraction so that the window range just covers the candidate interval, thereby improving the problem of high optical flow intensity caused by error accumulation.

[0139] In some embodiments, in step S108, video frame intervals are filtered using an empirical mode decomposition algorithm based on a third video frame candidate interval and a preset reconstruction threshold to obtain micro-expression detection intervals, including:

[0140] Step S601: Decompose the second optical flow intensity value of the candidate interval of the third video frame using the empirical mode decomposition algorithm to obtain the decomposed intrinsic mode function of the candidate interval of the third video frame.

[0141] Step S602: Remove the lowest frequency intrinsic mode function of the decomposed intrinsic mode function to obtain the removed intrinsic mode function of the candidate interval of the third video frame;

[0142] Step S603: Reconstruct the removed intrinsic mode functions to obtain the reconstructed signal of the candidate interval of the third video frame;

[0143] Step S604: Select video frame intervals whose reconstructed signals are greater than the preset reconstruction threshold from the candidate intervals of the third video frame to obtain the micro-expression detection interval.

[0144] Specifically, in some embodiments, reference is made to Figure 4 , Figure 4 In the EMD decomposition algorithm, "filter out low-frequency IMF" means removing the lowest-frequency IMF, "reconstruction" means reconstruction, "exceed" means exceeding, "filter out" means removal, "readjusted candidate interval" is the readjusted candidate interval, "output" is the output, and "pred onset (offset)" is the prediction start frame (end frame). Specifically, the optical flow signal (oflow2) extracted in the second stage is decomposed using the EMD method to obtain multiple IMFs. After removing the lowest-frequency IMF, the remaining IMFs are reconstructed to obtain a new signal (oflowh). By judging whether the reconstructed signal exceeds a set threshold, slow head movements and facial expressions are further identified. For candidate intervals that meet the conditions, the positions of their start and end frames are output as the prediction results.

[0145] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0146] I. Feature Point Detection:

[0147] Obtain the video to be tested.

[0148] Establish a sliding window and set the starting point of the sliding window at the position of the starting frame of the video to be detected. The sliding window covers a preset number of video frames.

[0149] Feature point detection is performed on the first reference frame to obtain several preset sub-regions within the face region. The first reference frame is the first video frame in the sliding window, and the first subsequent frames are all video frames in the sliding window except for the reference frame.

[0150] II. First-stage candidate range screening:

[0151] Calculate the optical flow intensity value of the nose tip region between each first subsequent frame and the first reference frame;

[0152] The first subsequent frame is aligned with the first reference frame based on the optical flow intensity value of the nose tip region to obtain the first aligned video frame;

[0153] Calculate the optical flow intensity values ​​of other sub-regions between the first aligned video frame and the first reference frame;

[0154] Calculate the average value of the optical flow intensity for each other sub-region, and use the average value as the region optical flow intensity value for each other sub-region;

[0155] High-frequency noise in the optical flow intensity value of each other sub-region is removed by fast Fourier transform and low-pass filter to obtain the first optical flow intensity value of the preset sub-region of each video frame.

[0156] Find the first video frame interval where the first optical flow intensity value is greater than a preset stage threshold and the number of video frames exceeds a preset number of video frames.

[0157] Find the first peak value in the first video frame interval;

[0158] Subtract the preset stage threshold one from the first peak value to obtain the first difference;

[0159] Find the second video frame interval where the first difference is greater than the preset stage threshold two;

[0160] Find the second peak in the second video frame interval, and take the video frame where the second peak is located as the first starting frame;

[0161] From the first starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the preset number of video frames from the first starting frame to the historical time, and take it as the first starting position of the second video frame range.

[0162] From the first starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the range of preset video frame number intervals, and take it as the first end position of the second video frame interval.

[0163] The first video frame candidate range is obtained based on the first start position and the first end position.

[0164] III. Second Stage Selection of Candidate Ranges:

[0165] Calculate the optical flow intensity value of the nose tip region between each video frame and the second reference frame within the candidate interval of the second video frame;

[0166] Based on the optical flow intensity value of the nose tip region, each video frame in the candidate interval of the second video frame is aligned with the second reference frame to obtain the second aligned video frame;

[0167] Calculate the optical flow intensity values ​​of other sub-regions between the second aligned video frame and the second reference frame, wherein the second video frame candidate interval is composed of the first video frame candidate interval and video frames extended to a predetermined number of future times, and the second reference frame is the first video frame of the first video frame candidate interval.

[0168] Calculate the average value of the optical flow intensity for each other sub-region, and use the average value as the region optical flow intensity value for each other sub-region;

[0169] High-frequency noise in the optical flow intensity value of each other sub-region is removed by fast Fourier transform and low-pass filter to obtain the second optical flow intensity value of the preset sub-region of each video frame.

[0170] Find the third video frame interval where the second optical flow intensity value is greater than a preset stage threshold and the number of video frames exceeds a preset number of video frames;

[0171] Find the third peak in the third video frame interval;

[0172] Subtract the preset stage threshold one from the third wave peak to obtain the second difference;

[0173] Find the fourth video frame interval where the second difference is greater than the preset stage threshold two;

[0174] Find the fourth peak in the fourth video frame interval, and take the video frame where the fourth peak is located as the second starting frame;

[0175] From the second starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the preset number of video frames from the historical time, and take it as the second starting position of the second video frame range.

[0176] From the first starting frame, find the first video frame within the preset number of video frames and the optical flow intensity value decreases less than the preset stage threshold three, and take it as the second end position of the second video frame interval.

[0177] The third video frame candidate range is obtained based on the second start position and the second end position.

[0178] IV. Facial Expression Detection:

[0179] The second optical flow intensity value of the candidate interval of the third video frame is decomposed by empirical mode decomposition algorithm to obtain the decomposed intrinsic mode function of the candidate interval of the third video frame.

[0180] The lowest frequency intrinsic mode function of the decomposed intrinsic mode function is removed to obtain the removed intrinsic mode function of the candidate interval of the third video frame.

[0181] The reconstructed intrinsic mode functions after removal are obtained to obtain the reconstructed signal of the candidate interval of the third video frame;

[0182] The video frame intervals whose reconstructed signals are greater than the preset reconstruction threshold are selected from the candidate intervals of the third video frame to obtain the micro-expression detection interval;

[0183] When the preset number of video frames is greater than or equal to the number of video frames in the video to be detected, the micro-expression detection interval is taken as the expression detection result of the video to be detected.

[0184] When the preset number of video frames is less than the number of video frames in the video to be detected, the sliding window is moved along the time axis of the video to be detected with a preset step size. Each time the sliding window is moved, steps S102 to S108 are repeated until the starting point of the sliding window is outside the video to be detected, and the micro-expression detection range of all sliding windows is taken as the expression detection result of the video to be detected.

[0185] Specifically, refer to Figure 5 , Figure 5 In the image, 'a' represents the intensity of the optical flow signal extracted by the sliding window. It can be seen that the failure to recover after an expression caused the subsequent optical flow to remain high, making it easily mistaken for an expression. Figure 5 Figure b shows the optical flow intensity signal extracted in the second stage. The optical flow extracted in the second stage reduces the accumulation of differences, thereby alleviating the problem of excessively high optical flow intensity and reducing false detections.

[0186] Specifically, refer to Figure 6 , Figure 6 The results show that even with a rapid increase in optical flow intensity caused by a real facial expression, the reconstructed signal still exhibits significant fluctuations.

[0187] Specifically, refer to Figure 7 , Figure 7This shows that the reconstructed signal becomes flat for slow head movements.

[0188] Specifically, referring to Table 1, which shows the experimental results of this application compared with the top three in the 2022 Microexpression Detection Challenge (MEGC2022Challenge), the results show that the solution of this application is more advantageous.

[0189] Table 1

[0190]

[0191] Additionally, refer to Figure 8 An embodiment of the present invention provides an expression detection system, including a data acquisition module 1100, a sliding window establishment module 1200, a feature point detection module 1300, a first optical flow intensity value calculation module 1400, a first video frame candidate interval filtering module 1500, a second optical flow intensity value calculation module 1600, a third video frame candidate interval filtering module 1700, and an expression detection module 1800, wherein:

[0192] The data acquisition module 1100 is used to acquire the video to be detected;

[0193] The sliding window creation module 1200 is used to create a sliding window and set the starting point of the sliding window at the position of the starting frame of the video to be detected. The sliding window covers a preset number of video frames.

[0194] The feature point detection module 1300 is used to perform feature point detection on the first reference frame to obtain several preset sub-regions within the face region. The first reference frame is the first video frame in the sliding window, and the first subsequent frame is all video frames in the sliding window except for the reference frame.

[0195] The first optical flow intensity value calculation module 1400 is used to calculate the first optical flow intensity value of a preset sub-region of each video frame based on the first reference frame;

[0196] The first video frame candidate interval filtering module 1500 is used to filter video frame intervals based on the first optical flow intensity value and the preset stage threshold to obtain the first video frame candidate interval.

[0197] The second optical flow intensity value calculation module 1600 is used to calculate the second optical flow intensity value of a preset sub-region of each video frame in the second video frame candidate interval according to the second reference frame. The second video frame candidate interval is composed of the first video frame candidate interval and video frames extended to a preset number of future times. The second reference frame is the first video frame of the first video frame candidate interval.

[0198] The third video frame candidate interval filtering module 1700 is used to filter video frame intervals based on the second optical flow intensity value and the preset stage threshold to obtain the third video frame candidate interval.

[0199] The expression detection module 1800 is used to filter video frame intervals based on the candidate intervals of the third video frame and the preset reconstruction threshold using the empirical mode decomposition algorithm to obtain the micro-expression detection intervals, and use the micro-expression detection intervals as the expression detection results of the video to be detected.

[0200] This system first establishes a sliding window, setting its starting point at the beginning frame of the video to be detected. The sliding window covers a preset number of video frames. Feature point detection is performed on a first reference frame to obtain several preset sub-regions within the face area. The first reference frame is the first video frame in the sliding window, and the first subsequent frames are all video frames in the sliding window except the reference frame. A first optical flow intensity value is calculated for each preset sub-region of each video frame based on the first reference frame. Video frame intervals are filtered based on the first optical flow intensity value and a preset stage threshold to obtain a first video frame candidate interval. A second optical flow intensity value is calculated for each preset sub-region of a video frame within the second video frame candidate interval based on a second reference frame. The second video frame candidate interval consists of the first video frame candidate interval and a preset number of video frames extended to future times. The second reference frame is the first video frame of the first video frame candidate interval. The system then calculates the second optical flow intensity value based on the second optical flow intensity value and a preset stage threshold. The application uses a two-stage optical flow strategy. First, a sliding window scheme is used to select intervals with higher optical flow intensity as candidate intervals. A separate reference frame is set for each candidate interval. Then, face alignment is performed again based on the new reference frame to reduce the influence of actions unrelated to facial expressions. At the same time, the window size for the second-stage optical flow extraction is dynamically adjusted so that the window range just covers the candidate interval, thereby improving the problem of high optical flow intensity caused by error accumulation. Then, based on the candidate intervals of the third video frame and the preset reconstruction threshold, the video frame intervals are selected using the empirical mode decomposition algorithm to obtain the micro-expression detection interval. The spectral analysis method is used to identify whether the interval contains facial expressions, which can better identify facial expressions and slow head movements. The micro-expression detection interval is used as the expression detection result of the video to be detected, reducing the number of false detections and improving the detection accuracy.

[0201] It should be noted that this system embodiment is based on the same inventive concept as the above system embodiment. Therefore, the relevant content of the above method embodiment is also applicable to this system embodiment, and will not be repeated here.

[0202] This application also provides an electronic device for facial expression detection, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the facial expression detection method as described above.

[0203] The processor and memory can be connected via a bus or other means.

[0204] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0205] The non-transient software program and instructions required to implement the expression detection method of the above embodiments are stored in memory. When executed by a processor, the expression detection method in the above embodiments is executed, for example, the method described above is executed. Figure 1 The method steps S101 to S108.

[0206] This application also provides a computer-readable storage medium storing computer-executable instructions for performing the facial expression detection method described above.

[0207] The computer-readable storage medium stores computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described electronic device embodiment, causing the processor to perform the facial expression detection method in the above-described embodiment, for example, to perform the above-described... Figure 1 The method steps S101 to S108.

[0208] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program units, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program units, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0209] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A facial expression detection method, characterized in that, The facial expression detection method includes: Obtain the video to be tested; A sliding window is established, and the starting point of the sliding window is set at the position of the starting frame of the video to be detected, wherein the sliding window covers a preset number of video frames; Feature point detection is performed on the first reference frame to obtain several preset sub-regions within the face region, wherein the first reference frame is the first video frame in the sliding window; Calculate the first optical flow intensity value of a preset sub-region of each video frame based on the first reference frame; Based on the first optical flow intensity value and the preset stage threshold, the video frame interval is filtered to obtain the first video frame candidate interval; The second optical flow intensity value of a preset sub-region of each video frame within the second video frame candidate interval is calculated based on the second reference frame. The second video frame candidate interval is composed of the first video frame candidate interval and video frames extended to a preset number of future times. The second reference frame is the first video frame of the first video frame candidate interval. Based on the second optical flow intensity value and the preset stage threshold, video frame intervals are filtered to obtain a third video frame candidate interval; Based on the candidate interval of the third video frame and the preset reconstruction threshold, the video frame interval is filtered by the empirical mode decomposition algorithm to obtain the micro-expression detection interval, and the micro-expression detection interval is used as the expression detection result of the video to be detected.

2. The facial expression detection method according to claim 1, characterized in that, The preset sub-region includes a nose tip sub-region and other sub-regions besides the nose tip sub-region. The step of calculating the first optical flow intensity value of the preset sub-region for each video frame based on the first reference frame includes: Calculate the optical flow intensity value of the nose tip region between each first subsequent frame and the first reference frame, wherein the first subsequent frame is all video frames in the sliding window except the reference frame; The first subsequent frame is aligned with the first reference frame based on the optical flow intensity value of the nose tip region to obtain the first aligned video frame; Calculate the optical flow intensity values ​​of the other sub-regions between the first aligned video frame and the first reference frame; Calculate the average value of the optical flow intensity value for each of the other sub-regions, and use the average value as the region optical flow intensity value corresponding to each of the other sub-regions; High-frequency noise in the optical flow intensity value corresponding to each of the other sub-regions is removed by low-pass filtering to obtain the first optical flow intensity value of the preset sub-region of each video frame.

3. The facial expression detection method according to claim 1, characterized in that, The preset stage thresholds include preset stage threshold one, preset stage threshold two, and preset stage threshold three. The step of filtering video frame intervals based on the first optical flow intensity value and the preset stage thresholds to obtain a first video frame candidate interval includes: Find the first video frame interval where the first optical flow intensity value is greater than the preset stage threshold and the number of video frames exceeds the preset number of video frames; Find the first peak value in the first video frame interval; Subtract the preset stage threshold from the first wave peak value to obtain the first difference; Find the second video frame interval where the first difference is greater than the preset stage threshold two; Find the second peak value in the second video frame interval, and take the video frame where the second peak value is located as the first starting frame; From the first starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the preset number of video frames from the first starting frame to the historical time, and take it as the first starting position of the second video frame range. From the first starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the range of the preset number of video frames, and take it as the first end position of the second video frame range. The first video frame candidate range is obtained based on the first start position and the first end position.

4. The facial expression detection method according to claim 3, characterized in that, The step of filtering video frame intervals based on the second optical flow intensity value and the preset stage threshold to obtain a third video frame candidate interval includes: Find a third video frame interval where the second optical flow intensity value is greater than the preset stage threshold and the number of video frames exceeds the preset number of video frames; Find the third peak value in the third video frame interval; Subtract the preset stage threshold from the third wave peak value to obtain the second difference; Find the fourth video frame interval where the second difference is greater than the preset stage threshold two; Find the fourth peak value in the fourth video frame interval, and take the video frame where the fourth peak value is located as the second starting frame; From the second starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the preset video frame number range, and take it as the second starting position of the second video frame range. From the first starting frame, find the first video frame in the range where the optical flow intensity value decreases less than the preset stage threshold three within the range of the preset number of video frames, and take it as the second end position of the second video frame range. The third video frame candidate range is obtained based on the second start position and the second end position.

5. The facial expression detection method according to claim 1, characterized in that, The preset sub-region includes a nose tip sub-region and other sub-regions besides the nose tip sub-region. The calculation of the second optical flow intensity value of the preset sub-region for each video frame within the second video frame candidate interval based on the second reference frame, wherein the second video frame candidate interval is composed of the first video frame candidate interval and video frames extended to a preset number of future times, and the second reference frame is the first video frame of the first video frame candidate interval, including: Calculate the optical flow intensity value of the nose tip region between each video frame in the second video frame candidate interval and the second reference frame; Based on the optical flow intensity value of the nose tip region, each video frame in the second video frame candidate interval is aligned with the second reference frame to obtain the second aligned video frame; Calculate the optical flow intensity values ​​of the other sub-regions between the second aligned video frame and the second reference frame; Calculate the average value of the optical flow intensity value for each of the other sub-regions, and use the average value as the region optical flow intensity value corresponding to each of the other sub-regions; High-frequency noise in the optical flow intensity value corresponding to each of the other sub-regions is removed by low-pass filtering to obtain the second optical flow intensity value of the preset sub-region of each video frame.

6. The facial expression detection method according to claim 1, characterized in that, The step of filtering video frame intervals using an empirical mode decomposition algorithm based on the third video frame candidate interval and a preset reconstruction threshold to obtain micro-expression detection intervals includes: The second optical flow intensity value of the candidate interval of the third video frame is decomposed by empirical mode decomposition algorithm to obtain the decomposed intrinsic mode function of the candidate interval of the third video frame. The lowest frequency intrinsic mode function of the decomposed intrinsic mode function is removed to obtain the removed intrinsic mode function of the candidate interval of the third video frame; The removed intrinsic mode functions are reconstructed to obtain the reconstructed signal of the candidate interval of the third video frame; The micro-expression detection interval is obtained by selecting video frame intervals whose reconstructed signals are greater than the preset reconstruction threshold.

7. The facial expression detection method according to claim 1, characterized in that, The expression detection method also includes: When the preset number of video frames is less than the number of video frames in the video to be detected, the sliding window is moved along the time axis of the video to be detected with a preset step size. Each time the sliding window is moved, the micro-expression detection interval of the current sliding window is calculated based on the first video frame of the current sliding window. This process is repeated until the starting point of the sliding window is outside the video to be detected. The micro-expression detection intervals of all sliding windows are then used as the expression detection result of the video to be detected.

8. An expression detection system, characterized in that, The facial expression detection system includes: The data acquisition module is used to acquire the video to be detected; A sliding window creation module is used to create a sliding window and set the starting point of the sliding window at the position of the starting frame of the video to be detected, wherein the sliding window covers a preset number of video frames; The feature point detection module is used to perform feature point detection on the first reference frame to obtain several preset sub-regions within the face region, wherein the first reference frame is the first video frame in the sliding window; The first optical flow intensity value calculation module is used to calculate the first optical flow intensity value of a preset sub-region of each video frame based on the first reference frame; The first video frame candidate interval filtering module is used to filter video frame intervals based on the first optical flow intensity value and a preset stage threshold to obtain the first video frame candidate interval. The second optical flow intensity value calculation module is used to calculate the second optical flow intensity value of a preset sub-region of each video frame in the second video frame candidate interval according to the second reference frame. The second video frame candidate interval is composed of the first video frame candidate interval and video frames extended to a preset number of future times. The second reference frame is the first video frame of the first video frame candidate interval. The third video frame candidate interval filtering module is used to filter video frame intervals based on the second optical flow intensity value and the preset stage threshold to obtain the third video frame candidate interval. The expression detection module is used to filter video frame intervals using an empirical mode decomposition algorithm based on the candidate interval of the third video frame and a preset reconstruction threshold to obtain micro-expression detection intervals, and to use the micro-expression detection intervals as the expression detection results of the video to be detected.

9. An expression detection device, characterized in that, The device includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform an expression detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform an expression detection method as described in any one of claims 1 to 7.

Images