A micro-expression detection method based on dual temporal stream network

Abstract

The present invention relates to the technical field of micro-expression detection, and specifically discloses a micro-expression detection method based on a dual temporal flow network. The method comprises: obtaining a face video to be detected and preprocessing a facial image sequence in the face video to be detected; extracting a first dense optical flow image sequence and a second dense optical flow image sequence from the preprocessed facial image sequence; sequentially inputting all identical dense optical flow images in the first dense optical flow image sequence and the second dense optical flow image sequence into a dual temporal flow network model for detection, thereby obtaining a micro-expression prediction value sequence corresponding to the facial image sequence; performing peak detection on the micro-expression prediction value sequence to obtain multiple predicted micro-expression intervals, and deduplicating the multiple predicted micro-expression intervals to obtain multiple target micro-expression intervals after deduplication; and outputting a micro-expression detection result for the face video to be detected based on the multiple target micro-expression intervals. The present invention can improve the accuracy of micro-expression detection.

Description

Technical Field

[0001] The present invention relates to the technical field of micro-expression detection, and more specifically, to a micro-expression detection method based on a dual temporal stream network. Background Art

[0002] Facial expressions are a crucial channel for interpersonal communication, and recognizing them can facilitate understanding of a person's psychological state and emotions. Unlike regular facial expressions, microexpressions are specialized, minute facial movements that can serve as an important indicator of subjective emotions, with wide-ranging applications in public security and psychotherapy. However, because microexpressions are characterized by small amplitude, short duration, and often interspersed with macroexpressions, manual detection of microexpressions requires specialized training and has a low accuracy rate.

[0003] In recent years, many researchers have begun to use computer vision technology to automatically locate (detect) micro-expressions, which has greatly improved the feasibility of micro-expressions. However, there are still some problems in the current research on micro-expression detection: (1) Existing micro-expression detection algorithms are easily affected by macro-expressions, and the false positive rate is relatively high. (2) Existing micro-expression detection algorithms are mostly based on single-time optical flow for feature extraction, and the label division is mostly based on the [0,1] hard division of the micro-expression interval, which is insufficient for the weaker action units in the initial part. (3) The post-processing methods for micro-expressions in existing research mostly use rough merging or NMS, without considering the length of different micro-expression intervals, resulting in low micro-expression detection accuracy. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a micro-expression detection method based on a dual temporal stream network to solve the problem of low micro-expression detection accuracy in the related art.

[0005] As a first aspect of the present invention, a micro-expression detection method based on a dual temporal stream network is provided, comprising the following steps:

[0006] Step S1: obtaining a face video to be detected, and preprocessing a face image sequence in the face video to be detected;

[0007] Step S2: extracting a first dense optical flow image sequence and a second dense optical flow image sequence from the pre-processed face image sequence respectively;

[0008] Step S3: inputting all identical-frame dense optical flow images in the first dense optical flow image sequence and the second dense optical flow image sequence into a dual temporal flow network model for detection, and obtaining a micro-expression prediction value sequence corresponding to the facial image sequence, wherein the micro-expression prediction value sequence includes a micro-expression prediction value for each frame of the facial image in the facial image sequence;

[0009] Step S4: performing peak detection on the micro-expression prediction value sequence to obtain multiple predicted micro-expression intervals in the face video to be detected, and removing duplicates from the multiple predicted micro-expression intervals to obtain multiple target micro-expression intervals after removal of duplicates;

[0010] Step S5: Outputting a micro-expression detection result of the face video to be detected according to the multiple target micro-expression intervals.

[0011] Furthermore, the step of obtaining a face video to be detected and preprocessing a face image sequence in the face video to be detected includes:

[0012] According to the face detection tool, the face image sequence S in the face video to be detected is calculated as follows: i ,…,src N Perform face detection frame by frame and obtain the face coordinate frame bbox={bbox1,…,bbox i ,…,bbox N} and facial key points lmk = {lmk1,…,lmk i ,…,lmk N}, where N is the total number of frames of the face video to be detected, src i is the face image of frame i in the face video to be detected, bbox i is the face coordinate frame of the face image of the i-th frame in the face video to be detected, lmk i is the facial key point of the i-th face image in the face video to be detected;

[0013] Normalize the face images in the face image sequence S according to the face coordinate frame and the face key points to obtain a corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f}.

[0014] Furthermore, the step of respectively extracting a first dense optical flow image sequence and a second dense optical flow image sequence from the preprocessed facial image sequence includes:

[0015] Get the first step length K for extracting dense optical flow images Slow and the second step length K Fast ;

[0016] According to the first step length K Slow , extract the corrected face image sequence S f ={src 1-f,…,src i-f ,…,src N-f} in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Slow The first dense optical flow image sequence F between the face images after frame correction Slow ={f 1-Slow ,…,f i-Slow ,…,f N-Slow}where f i-Slow is the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f} in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Slow The dense optical flow image between the face images after frame correction, f i-Slow That is the first dense optical flow image sequence F Slow The i-th frame dense optical flow image in;

[0017] According to the second step length K Fast , extract the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f} in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Fast The second dense optical flow image sequence F between the face images after frame correction Fast ={f 1-Fast ,…,f i-Fast ,…,f N-Fast}where f i-Fast is the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f} in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Fast The dense optical flow image between the face images after frame correction, f i-Fast That is the second dense optical flow image sequence F Fast The i-th frame dense optical flow image in .

[0018] Furthermore, the first step length K Slow =1, the second step length K Fast The calculation formula is shown in (1)

[0019]

[0020] Wherein, T is the number of target micro-expression intervals in the face video to be detected, nj-offset -n j-onset is the length of the jth target micro-expression interval, n j-offset is the end frame of the jth target micro-expression interval, n j-onset is the starting frame of the jth target micro-expression interval.

[0021] Furthermore, all identical-frame dense optical flow images in the first dense optical flow image sequence and the second dense optical flow image sequence are sequentially input into the dual temporal flow network model for detection to obtain a micro-expression prediction value sequence corresponding to the facial image sequence, wherein the micro-expression prediction value sequence includes the micro-expression prediction value of each frame of the facial image in the facial image sequence, including:

[0022] The dual temporal flow network model includes a first temporal flow backbone network, a second temporal flow backbone network, a Concat layer, a Linear layer, and a Sigmoid function;

[0023] Read the first dense optical flow image sequence F Slow The optical flow component of each dense optical flow image in the image, and read the second dense optical flow image sequence F Fast The optical flow component of each frame of dense optical flow image;

[0024] The first dense optical flow image sequence F Slow The optical flow component of the dense optical flow image of the i-th frame is input into the first temporal flow backbone network to output the i-th frame dense optical flow image f i-Slow Optical flow features;

[0025] The second dense optical flow image sequence F Fast The optical flow component of the dense optical flow image of the i-th frame is input into the second temporal flow backbone network to output the i-th dense optical flow image f i-Fast Optical flow features;

[0026] The i-th frame dense optical flow image f i-Slow The optical flow features and the dense optical flow image f of the i-th frame i-Fast The optical flow features of are merged through the Concat layer to output the merged optical flow features of the i-th frame face image;

[0027] The merged optical flow features of the i-th frame facial image are subjected to dimensionality reduction processing by the Linear layer to obtain reduced-dimensional optical flow features, and then the reduced-dimensional optical flow features are output through the Sigmoid function to output the micro-expression prediction value of the i-th frame facial image in the facial image sequence.

[0028] Furthermore, it also includes:

[0029] The first dense optical flow image sequence F Slow The optical flow component of the dense optical flow image of the i-th frame and the second dense optical flow image sequence F Fast The optical flow component of the dense optical flow image of the i-th frame passes through the same temporal flow backbone network. For each temporal flow backbone network, a dense optical flow image of the i-th frame with a size of 1×112×112 is input and its f x ,f y The optical flow features in the two directions are respectively passed through Conv2d and MaxPool2d to obtain the optical flow features of size 1×37×37, and then f x ,f y The 1×37×37 optical flow features in two directions are merged into 2×37×37 optical flow features, which are passed through MaxPool2d to obtain optical flow features of size 2×18×18, and then flattened to obtain 1×648 optical flow image depth features. The 1×648 optical flow image depth features output by the first temporal stream backbone network and the 1×648 optical flow image depth features output by the second temporal stream backbone network are merged along the feature direction to obtain 1×1296 optical flow image depth features, and then connected to the Linear layer for dimensionality reduction processing to obtain 1×256 optical flow image depth features. Finally, the micro-expression prediction value of the i-th frame face image in the face image sequence is output through the Sigmoid function.

[0030] Furthermore, the peak detection of the micro-expression prediction value sequence is performed to obtain a plurality of predicted micro-expression intervals in the face video to be detected, including:

[0031] According to the empirical threshold and the second step length K Fast The peak value of the micro-expression prediction value sequence is detected by interval to obtain the peak point set P of the micro-expression prediction value corresponding to the face video to be detected = {p0, ..., p i ,…,p C}, p i is the i-th micro-expression prediction value peak point in the face video to be detected, and C is the total number of micro-expression prediction value peak points in the face video to be detected;

[0032] According to the peak point p of the micro-expression prediction value of the i-th micro-expression in the face video to be detected i and the second step length K Fast Get the i-th predicted micro-expression interval B in the face video to be detected i =[p i -K Fast ,p i+ K Fast ];

[0033] The face video to be detected includes C predicted micro-expression intervals.

[0034] Furthermore, the step of removing duplicates from the plurality of predicted micro-expression intervals to obtain a plurality of target micro-expression intervals after removing duplicates further includes:

[0035] According to the micro-expression prediction values ​​of the start frame and the end frame of each predicted micro-expression interval, the average prediction value of each predicted micro-expression interval is obtained;

[0036] selecting, based on average predicted values ​​of a plurality of predicted micro-expression intervals, a predicted micro-expression interval M with the largest average predicted value from the plurality of predicted micro-expression intervals, wherein the plurality of predicted micro-expression intervals consists of the predicted micro-expression interval M with the largest average predicted value and other N predicted micro-expression intervals;

[0037] Calculating the intersection-over-union ratio and the center-point distance of each predicted micro-expression interval in the other N predicted micro-expression intervals with the predicted micro-expression interval M;

[0038] updating the average predicted value of each of the other N predicted micro-expression intervals according to the intersection-over-union ratio and the distance between the centers of each predicted facial micro-expression interval and the predicted micro-expression interval M;

[0039] Among them, the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i Updated average prediction value s i The calculation formula is shown in (2);

[0040]

[0041] Among them, IoU is the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i The intersection-over-union ratio of the predicted micro-expression interval M, d1 is the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i and the total length of the predicted micro-expression interval M, d2 is the length of the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i The distance between the center point and the center point of the predicted micro-expression interval M;

[0042] The updated average prediction value s of all the predicted micro-expression intervals is i The predicted micro-expression intervals with a value of 0 are deleted to obtain the multiple target micro-expression intervals after duplicate removal.

[0043] Furthermore, the loss functions of the first time sequence flow backbone network and the second time sequence flow backbone network both adopt MSE loss, and the final loss function l of the dual time sequence flow network model is shown in formula (3);

[0044]

[0045] Wherein, N is the first dense optical flow image sequence F Slow Or the second dense optical flow image sequence F Fast The total number of image frames in y i is the micro-expression prediction value of the i-th frame face image, label i-Slow is the first dense optical flow image sequence F Slow The label of the dense optical flow image of the i-th frame, label i-Fast is the second dense optical flow image sequence F Fast The label of the dense optical flow image of the i-th frame, the coefficient of λ∈[0,1].

[0046] Furthermore, it also includes:

[0047] When the first dense optical flow image sequence F Slow When the dense optical flow image of the i-th frame falls into the target micro-expression interval, its label label i-Slow is 1, otherwise label label i-Slow is 0;

[0048] When the second dense optical flow image sequence F fast When the dense optical flow image of the i-th frame falls into the target micro-expression interval, its label label i-Fast The calculation formula is shown in (4), otherwise the label label i-Fast is 0;

[0049]

[0050] Among them, onset, apex and offset are the starting frame, peak frame and end frame of the target micro-expression interval respectively.

[0051] The present invention provides a micro-expression detection method based on a dual temporal flow network, which has the following advantages: the micro-expression detection task is achieved by applying two temporal optical flow feature learning networks and their corresponding labels (Slow and Fast). At the same time, redundant overlapping micro-expression intervals are eliminated based on one-dimensional DIoU-NMS, thereby improving the accuracy of micro-expression detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0053] Figure 1 The present invention provides a flowchart of a micro-expression detection method based on a dual temporal stream network.

[0054] Figure 2 This is a structural framework diagram of the dual-time sequence flow network model provided by the present invention.

[0055] Figure 3 Schematic diagram of the intersection of two predicted micro-expression intervals provided by the present invention.

[0056] Figure 4 This is a schematic diagram of the division of optical flow image labels in the first dense optical flow image sequence and the optical flow image labels in the second dense optical flow image sequence provided by the present invention. DETAILED DESCRIPTION

[0057] To further illustrate the technical means and effectiveness of the present invention in achieving its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of a micro-expression detection method based on a dual temporal stream network. It should be understood that the described embodiments are only a subset of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0058] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present invention described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0059] In this embodiment, a micro-expression detection method based on a dual temporal flow network is provided. Figure 1 As shown, a micro-expression detection method based on a dual temporal stream network includes the following steps:

[0060] Step S1: obtaining a face video to be detected, and preprocessing a face image sequence in the face video to be detected;

[0061] Step S2: extracting a first dense optical flow image sequence and a second dense optical flow image sequence from the pre-processed face image sequence respectively;

[0062] Step S3: inputting all identical-frame dense optical flow images in the first dense optical flow image sequence and the second dense optical flow image sequence into a dual temporal flow network model for detection, and obtaining a micro-expression prediction value sequence corresponding to the facial image sequence, wherein the micro-expression prediction value sequence includes a micro-expression prediction value for each frame of the facial image in the facial image sequence;

[0063] Step S4: performing peak detection on the micro-expression prediction value sequence to obtain multiple predicted micro-expression intervals in the face video to be detected, and removing duplicates from the multiple predicted micro-expression intervals to obtain multiple target micro-expression intervals after removal of duplicates;

[0064] Step S5: Outputting a micro-expression detection result of the face video to be detected according to the multiple target micro-expression intervals.

[0065] Preferably, the step of obtaining a face video to be detected and preprocessing a face image sequence in the face video to be detected includes:

[0066] According to the face detection tool, the face image sequence S in the face video to be detected is calculated as follows: i ,…,src N Perform face detection frame by frame and obtain the face coordinate frame bbox={bbox1,…,bbox i ,…,bbox N} and facial key points lmk = {lmk1,…,lmk i ,…,lmk N}, where N is the total number of frames of the face video to be detected, src i is the face image of frame i in the face video to be detected, bbox i is the face coordinate frame of the face image of the i-th frame in the face video to be detected, lmk i is the facial key point of the i-th face image in the face video to be detected;

[0067] Specifically, the face detection tool may include RetinaFace (Single-stage Dense Face Localisation in the Wild).

[0068] Normalize the face images in the face image sequence S according to the face coordinate frame and the face key points to obtain a corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f}.

[0069] In order to reduce the impact of camera shaking or face swing, all faces need to be normalized to the same scale and frontal angle, based on the facial key points lmk i Perform 5-point face alignment to obtain the corrected face image

[0070] In the embodiment of the present invention, the corrected face image S is extracted. f TV-L1 dense optical flow Used to represent the action changes of expression units, [f i x ,f i y ] is the optical flow component of the dense optical flow of frame i in the x / y direction.

[0071] Preferably, the step of respectively extracting a first dense optical flow image sequence and a second dense optical flow image sequence from the pre-processed facial image sequence comprises:

[0072] Get the first step length K for extracting dense optical flow images Slow and the second step length K Fast ;

[0073] According to the first step length K Slow , extract the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f} in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Slow The first dense optical flow image sequence F between the face images after frame correction Slow ={f 1-Slow ,…,f i-Slow ,…,f N-Slow}where f i-Slow is the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f} in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Slow The dense optical flow image between the face images after frame correction, f i-SlowThat is the first dense optical flow image sequence F Slow The i-th frame dense optical flow image in;

[0074] According to the second step length K Fast , extract the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f} in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Fast The second dense optical flow image sequence F between the face images after frame correction Fast ={f 1-Fast ,…,f i-Fast ,…,f N-Fast}where f i-Fast is the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f} in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Fast The dense optical flow image between the face images after frame correction, f i-Fast That is the second dense optical flow image sequence F Fast The i-th frame dense optical flow image in .

[0075] In the embodiment of the present invention, the first step length K Slow =1, the second step length K Fast The calculation formula is shown in (1)

[0076]

[0077] Wherein, T is the number of target micro-expression intervals in the face video to be detected, n j-offset -n j-onset is the length of the jth target micro-expression interval, n j-offset is the end frame of the jth target micro-expression interval, n j-onset is the starting frame of the jth target micro-expression interval.

[0078] It should be noted that the second step length K Fast It is half of the average length of the target micro-expression interval in the dataset.

[0079] Preferably, if Figure 2As shown, all identical-frame dense optical flow images in the first dense optical flow image sequence and the second dense optical flow image sequence are sequentially input into the dual temporal flow network model for detection to obtain a micro-expression prediction value sequence corresponding to the facial image sequence, wherein the micro-expression prediction value sequence includes the micro-expression prediction value of each frame of the facial image in the facial image sequence, including:

[0080] The dual temporal flow network model includes a first temporal flow backbone network, a second temporal flow backbone network, a Concat layer, a Linear layer, and a Sigmoid function;

[0081] Read the first dense optical flow image sequence F Slow The optical flow component of each dense optical flow image in the image, and read the second dense optical flow image sequence F Fast The optical flow component of each frame of dense optical flow image;

[0082] The first dense optical flow image sequence F Slow The optical flow component of the dense optical flow image of the i-th frame is input into the first temporal flow backbone network to output the i-th frame dense optical flow image f i-Slow Optical flow features;

[0083] The second dense optical flow image sequence F Fast The optical flow component of the dense optical flow image of the i-th frame is input into the second temporal flow backbone network to output the i-th dense optical flow image f i-Fast Optical flow features;

[0084] The i-th frame dense optical flow image f i-Slow The optical flow features and the dense optical flow image f of the i-th frame i-Fast The optical flow features of are merged through the Concat layer to output the merged optical flow features of the i-th frame face image;

[0085] The merged optical flow features of the i-th frame facial image are subjected to dimensionality reduction processing by the Linear layer to obtain reduced-dimensional optical flow features, and then the reduced-dimensional optical flow features are output through the Sigmoid function to output the micro-expression prediction value of the i-th frame facial image in the facial image sequence.

[0086] Specifically, if Figure 2 As shown, it also includes:

[0087] The first dense optical flow image sequence F Slow The optical flow component of the dense optical flow image of the i-th frame and the second dense optical flow image sequence F FastThe optical flow component of the dense optical flow image of the i-th frame passes through the same temporal flow backbone network. For each temporal flow backbone network, a dense optical flow image of the i-th frame with a size of 1×112×112 is input and its f x ,f y The optical flow features in the two directions are respectively passed through Conv2d and MaxPool2d to obtain the optical flow features of size 1×37×37, and then f x ,f y The 1×37×37 optical flow features in two directions are merged into 2×37×37 optical flow features, which are then subjected to MaxPool2d to obtain optical flow features of size 2×18×18. Flattening is then performed to obtain 1×648 optical flow image depth features. The 1×648 optical flow image depth features output by the first temporal stream backbone network and the 1×648 optical flow image depth features output by the second temporal stream backbone network are merged along the feature direction to obtain 1×1296 optical flow image depth features. The features are then connected to the Linear layer for dimensionality reduction to obtain 1×256 optical flow image depth features. Finally, the micro-expression prediction value of the i-th frame of the facial image in the facial image sequence is output through the Sigmoid function.

[0088] Preferably, the step of performing peak detection on the micro-expression prediction value sequence to obtain a plurality of predicted micro-expression intervals in the face video to be detected comprises:

[0089] The micro-expression prediction values ​​of the dual temporal stream network model are smoothed and denoised to eliminate the impact caused by the prediction deviation of the dual temporal stream network model.

[0090] According to the empirical threshold and the second step length K Fast The peak value of the micro-expression prediction value sequence is detected by interval to obtain the peak point set P of the micro-expression prediction value corresponding to the face video to be detected = {p0, ..., p i ,…,p C}, p i is the i-th micro-expression prediction value peak point in the face video to be detected, and C is the total number of micro-expression prediction value peak points in the face video to be detected;

[0091] According to the peak point p of the micro-expression prediction value of the i-th micro-expression in the face video to be detected i and the second step length K Fast Get the i-th predicted micro-expression interval B in the face video to be detected i =[p i -K Fast ,p i+ K Fast ];

[0092] The face video to be detected includes C predicted micro-expression intervals.

[0093] The predicted micro-expression intervals are affected by the deviation of the dual temporal flow network model and the peak detection, and often have large redundancy (such as overlap, etc.). Traditional methods use the 1D-NMS method to eliminate redundancy, and IoU is the only factor it considers. When there is an order of magnitude difference in the length of the two overlapping expression intervals, this method will fail. Comprehensively consider the overlap of the intersecting expression intervals and the distance between the center points, such as Figure 3 As shown, 1D-DIoU-NMS (one-dimensional distance intersection over union (IOU) method) is used as a processing method to eliminate redundancy. Figure 3 It can be seen that when two predicted micro-expression intervals are respectively in the case of intersecting but not containing or containing each other, the corresponding expression intervals can be ideally deleted or retained.

[0094] Specifically, removing duplicates from the plurality of predicted micro-expression intervals to obtain a plurality of target micro-expression intervals after removing duplicates further includes:

[0095] According to the micro-expression prediction values ​​of the start frame and the end frame of each predicted micro-expression interval, the average prediction value of each predicted micro-expression interval is obtained;

[0096] selecting, based on average predicted values ​​of a plurality of predicted micro-expression intervals, a predicted micro-expression interval M with the largest average predicted value from the plurality of predicted micro-expression intervals, wherein the plurality of predicted micro-expression intervals consists of the predicted micro-expression interval M with the largest average predicted value and other N predicted micro-expression intervals;

[0097] Calculating the intersection-over-union ratio and the center-point distance of each predicted micro-expression interval in the other N predicted micro-expression intervals with the predicted micro-expression interval M;

[0098] updating the average predicted value of each of the other N predicted micro-expression intervals according to the intersection-over-union ratio and the distance between the centers of each predicted facial micro-expression interval and the predicted micro-expression interval M;

[0099] Among them, the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i Updated average prediction value s i The calculation formula is shown in (2);

[0100]

[0101] Among them, IoU is the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals iThe intersection-over-union ratio of the predicted micro-expression interval M, d1 is the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i and the total length of the predicted micro-expression interval M, d2 is the length of the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i The distance between the center point and the center point of the predicted micro-expression interval M;

[0102] The updated average prediction value s of all the predicted micro-expression intervals is i The predicted micro-expression intervals with a value of 0 are deleted to obtain the multiple target micro-expression intervals after duplicate removal.

[0103] In the embodiment of the present invention, the loss functions of the first time sequence flow backbone network and the second time sequence flow backbone network both adopt MSE loss, and the final loss function l of the dual time sequence flow network model is shown in formula (3);

[0104]

[0105] Wherein, N is the first dense optical flow image sequence F Slow Or the second dense optical flow image sequence F Fast The total number of image frames in y i is the micro-expression prediction value of the i-th frame face image, label i-Slow is the first dense optical flow image sequence F Slow The label of the dense optical flow image of the i-th frame, label i-Fast is the second dense optical flow image sequence F Fast The label of the dense optical flow image of the i-th frame, the coefficient of λ∈[0,1].

[0106] Due to optical flow aperture issues, we can't treat the temporal and spatial dimensions symmetrically. At a given moment in the world we typically see, the world is generally static, and spatial semantic information is typically constant or extremely slow. However, modeling micro-expressions requires fast frame rates to reflect the state changes of the action. We mimic the retinal neural operation of primate vision, capturing both spatial and temporal semantic information in video at both slow and fast frame rates to improve micro-expression detection.

[0107] Specifically, if Figure 4 As shown, it also includes:

[0108] When the first dense optical flow image sequence F Slow When the dense optical flow image of the i-th frame falls into the target micro-expression interval, its label label i-Slow is 1, otherwise label labeli-Slow is 0;

[0109] When the second dense optical flow image sequence F fast When the dense optical flow image of the i-th frame falls into the target micro-expression interval, its label label i-Fast The calculation formula is shown in (4), otherwise the label label i-Fast is 0;

[0110]

[0111] Among them, onset, apex and offset are the starting frame, peak frame and end frame of the target micro-expression interval respectively.

[0112] In summary, the present invention provides a micro-expression detection method based on a dual temporal flow regression network in long videos. The micro-expression detection task is achieved by applying two temporal optical flow feature learning networks and their corresponding labels (Slow and Fast). At the same time, redundant overlapping micro-expression intervals are eliminated based on one-dimensional DIoU-NMS, thereby improving the accuracy of micro-expression detection.

[0113] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the present profession can make slight changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A micro-expression detection method based on a dual temporal stream network, characterized in that: The steps include: Step S1: obtaining a face video to be detected, and preprocessing a face image sequence in the face video to be detected; Step S2: extracting a first dense optical flow image sequence and a second dense optical flow image sequence from the preprocessed face image sequence respectively; Step S3: inputting all identical-frame dense optical flow images in the first dense optical flow image sequence and the second dense optical flow image sequence into a dual temporal flow network model for detection, and obtaining a micro-expression prediction value sequence corresponding to the facial image sequence, wherein the micro-expression prediction value sequence includes a micro-expression prediction value for each frame of the facial image in the facial image sequence; Step S4: performing peak detection on the micro-expression prediction value sequence to obtain multiple predicted micro-expression intervals in the face video to be detected, and removing duplicates from the multiple predicted micro-expression intervals to obtain multiple target micro-expression intervals after removal of duplicates; Step S5: Outputting a micro-expression detection result of the face video to be detected according to the multiple target micro-expression intervals.

2. The micro-expression detection method based on a dual temporal stream network according to claim 1, characterized in that: The step of obtaining a face video to be detected and preprocessing a face image sequence in the face video to be detected includes: According to the face detection tool, the face image sequence S in the face video to be detected is calculated as follows: i ,…,src N Perform face detection frame by frame and obtain the face coordinate frame bbox={bbox1,…,bbox i ,…,bbox N } and facial key points lmk = {lmk1,…,lmk i ,…,lmk N }, where N is the total number of frames of the face video to be detected, src i is the face image of frame i in the face video to be detected, bbox i is the face coordinate frame of the face image of the i-th frame in the face video to be detected, lmk i is the facial key point of the i-th face image in the face video to be detected; Normalize the face images in the face image sequence S according to the face coordinate frame and the face key points to obtain a corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f }.

3. The micro-expression detection method based on a dual temporal stream network according to claim 2, characterized in that: The step of respectively extracting a first dense optical flow image sequence and a second dense optical flow image sequence from the pre-processed facial image sequence comprises: Get the first step length K for extracting dense optical flow images Slow and the second step length K Fast ; According to the first step length K Slow , extract the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f } in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Slow The first dense optical flow image sequence F between the face images after frame correction Slow ={f 1-Slow ,…,f i-Slow ,…,f N-Slow }where f i-Slow is the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f } in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Slow The dense optical flow image between the face images after frame correction, f i-Slow That is the first dense optical flow image sequence F Slow The i-th frame dense optical flow image in; According to the second step length K Fast , extract the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f } in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Fast The second dense optical flow image sequence F between the face images after frame correction Fast ={f 1-Fast ,…,f i-Fast ,…,f N-Fast }where f i-Fast is the corrected face image sequence S f ={src 1-f ,…,src i-f ,…,src N-f } in the i-th frame after correction of the face image and the i+K-th frame after correction of the face image Fast The dense optical flow image between the face images after frame correction, f i-Fast That is the second dense optical flow image sequence F Fast The i-th frame dense optical flow image in .

4. The micro-expression detection method based on a dual temporal stream network according to claim 3, characterized in that: First step length K Slow =1, the second step length K Fast The calculation formula is shown in (1) Wherein, T is the number of target micro-expression intervals in the face video to be detected, n j-offset -n j-onset is the length of the jth target micro-expression interval, n j-offset is the end frame of the jth target micro-expression interval, n j-onset is the starting frame of the jth target micro-expression interval.

5. The micro-expression detection method based on a dual temporal stream network according to claim 3, characterized in that: All identical-frame dense optical flow images in the first dense optical flow image sequence and the second dense optical flow image sequence are sequentially input into the dual temporal flow network model for detection to obtain a micro-expression prediction value sequence corresponding to the facial image sequence, wherein the micro-expression prediction value sequence includes a micro-expression prediction value for each frame of the facial image in the facial image sequence, including: The dual temporal flow network model includes a first temporal flow backbone network, a second temporal flow backbone network, a Concat layer, a Linear layer, and a Sigmoid function; Read the first dense optical flow image sequence F Slow The optical flow component of each dense optical flow image in the image, and read the second dense optical flow image sequence F Fast The optical flow component of each frame of dense optical flow image; The first dense optical flow image sequence F Slow The optical flow component of the dense optical flow image of the i-th frame is input into the first temporal flow backbone network to output the i-th frame dense optical flow image f i-Slow Optical flow features; The second dense optical flow image sequence F Fast The optical flow component of the dense optical flow image of the i-th frame is input into the second temporal flow backbone network to output the i-th dense optical flow image f i-Fast Optical flow features; The i-th frame dense optical flow image f i-Slow The optical flow features and the dense optical flow image f of the i-th frame i-Fast The optical flow features of are merged through the Concat layer to output the merged optical flow features of the i-th frame face image; The merged optical flow features of the i-th frame facial image are subjected to dimensionality reduction processing by the Linear layer to obtain reduced-dimensional optical flow features, and then the reduced-dimensional optical flow features are output through the Sigmoid function to output the micro-expression prediction value of the i-th frame facial image in the facial image sequence.

6. The micro-expression detection method based on a dual temporal stream network according to claim 5, characterized in that: Also includes: The first dense optical flow image sequence F Slow The optical flow component of the dense optical flow image of the i-th frame and the second dense optical flow image sequence F Fast The optical flow component of the dense optical flow image of the i-th frame passes through the same temporal flow backbone network. For each temporal flow backbone network, a dense optical flow image of the i-th frame with a size of 1×112×112 is input and its f x ,f y The optical flow features in the two directions are respectively passed through Conv2d and MaxPool2d to obtain the optical flow features of size 1×37×37, and then f x ,f y The 1×37×37 optical flow features in two directions are merged into 2×37×37 optical flow features, which are passed through MaxPool2d to obtain optical flow features of size 2×18×18, and then flattened to obtain 1×648 optical flow image depth features. The 1×648 optical flow image depth features output by the first temporal stream backbone network and the 1×648 optical flow image depth features output by the second temporal stream backbone network are merged along the feature direction to obtain 1×1296 optical flow image depth features, and then connected to the Linear layer for dimensionality reduction processing to obtain 1×256 optical flow image depth features. Finally, the micro-expression prediction value of the i-th frame face image in the face image sequence is output through the Sigmoid function.

7. The micro-expression detection method based on a dual temporal stream network according to claim 3, characterized in that: The step of performing peak detection on the micro-expression prediction value sequence to obtain a plurality of predicted micro-expression intervals in the face video to be detected includes: According to the empirical threshold and the second step length K Fast The peak value of the micro-expression prediction value sequence is detected by interval to obtain the peak point set P of the micro-expression prediction value corresponding to the face video to be detected = {p0, ..., p i ,…,p C }, p i is the i-th micro-expression prediction value peak point in the face video to be detected, and C is the total number of micro-expression prediction value peak points in the face video to be detected; According to the peak point p of the micro-expression prediction value of the i-th micro-expression in the face video to be detected i and the second step length K Fast Get the i-th predicted micro-expression interval B in the face video to be detected i =[p i -K Fast ,p i +K Fast ]; The face video to be detected includes C predicted micro-expression intervals.

8. The micro-expression detection method based on a dual temporal stream network according to claim 7, characterized in that: The step of removing duplicates from the plurality of predicted micro-expression intervals to obtain a plurality of target micro-expression intervals after removing duplicates further includes: According to the micro-expression prediction values ​​of the start frame and the end frame of each predicted micro-expression interval, the average prediction value of each predicted micro-expression interval is obtained; selecting, based on average predicted values ​​of a plurality of predicted micro-expression intervals, a predicted micro-expression interval M having the largest average predicted value from the plurality of predicted micro-expression intervals, wherein the plurality of predicted micro-expression intervals consists of the predicted micro-expression interval M having the largest average predicted value and other N predicted micro-expression intervals; Calculating the intersection-over-union ratio and the center-point distance of each predicted micro-expression interval in the other N predicted micro-expression intervals with the predicted micro-expression interval M; updating the average predicted value of each of the other N predicted micro-expression intervals according to the intersection-over-union ratio and the distance between the centers of each predicted facial micro-expression interval and the predicted micro-expression interval M; Among them, the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i Updated average prediction value s i The calculation formula is shown in (2); Among them, IoU is the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i The intersection-over-union ratio of the predicted micro-expression interval M, d1 is the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i and the total length of the predicted micro-expression interval M, d2 is the length of the i-th predicted micro-expression interval B among the other N predicted micro-expression intervals i The distance between the center point and the center point of the predicted micro-expression interval M; The updated average prediction value s of all the predicted micro-expression intervals is i The predicted micro-expression intervals with a value of 0 are deleted to obtain the multiple target micro-expression intervals after duplicate removal.

9. The micro-expression detection method based on a dual temporal stream network according to claim 5, characterized in that: The loss functions of the first time sequence flow backbone network and the second time sequence flow backbone network both adopt MSE loss, and the final loss function l of the dual time sequence flow network model is shown in formula (3); Wherein, N is the first dense optical flow image sequence F Slow Or the second dense optical flow image sequence F Fast The total number of image frames in y i is the micro-expression prediction value of the face image in frame i, label i-Slow is the first dense optical flow image sequence F Slow The label of the dense optical flow image of the i-th frame, label i-Fast is the second dense optical flow image sequence F Fast The label of the dense optical flow image of the i-th frame, the coefficient of λ∈[0,1].

10. The micro-expression detection method based on a dual temporal stream network according to claim 9, characterized in that: Also includes: When the first dense optical flow image sequence F Slow When the dense optical flow image of the i-th frame falls into the target micro-expression interval, its label label i-Slow is 1, otherwise label label i-Slow is 0; When the second dense optical flow image sequence F fast When the dense optical flow image of the i-th frame falls into the target micro-expression interval, its label label i-Fast The calculation formula is shown in (4), otherwise the label label i-Fast is 0; Among them, onset, apex and offset are the starting frame, peak frame and end frame of the target micro-expression interval respectively.

Images