Expression detection method, system, device and storage medium

Abstract

The present invention discloses an expression detection method, system, device and storage medium, which relate to the field of computer technology. The expression detection method includes obtaining a video to be detected; extracting optical flow features of the video to be detected; inputting the optical flow features of the video to be detected into an expression detection model built based on a Swin Transformer network to obtain first expression detection values ​​for all frames in the video; taking frames corresponding to first expression detection values ​​greater than a first threshold as candidate frames; determining multiple maximum points based on the first expression detection values ​​of all candidate frames; determining the frame with the largest expression amplitude based on the frame distance between the frames corresponding to the two maximum points, and determining the frame interval with expression based on the frame with the largest expression amplitude. The present application can improve the accuracy of video expression detection.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an expression detection method, system, device and storage medium. Background Art

[0002] Facial expressions, body language, and voice can reflect true human emotions, with facial expressions being a key way to express human emotion. Facial expressions are categorized into macro-expressions and micro-expressions. Macro-expressions occur automatically, can be faked, and have a large range of motion. Micro-expressions, on the other hand, are short-lived, have smaller movements, and occur subconsciously. They represent a person's attempt to suppress their true feelings and can reveal a person's true emotions. Analyzing micro-expressions can uncover underlying psychological processes.

[0003] Video expression analysis involves two parts: detection and recognition. Expression detection involves determining the presence of an expression in an expression video and locating its temporal location. Expression detection, particularly micro-expressions, is challenging to analyze manually due to their small movements and rapid changes. Therefore, developing a reliable, automated micro-expression analysis system is essential. With the advancement of computer vision technology, researchers have combined relevant algorithms to capture the dynamic characteristics of expressions for expression detection. With the outstanding performance of convolutional neural networks (CNNs) in various fields, many researchers have applied them to expression detection. Researchers first preprocess the expression image data to extract features. They then transform expression detection into an image classification problem, using CNNs to classify the image features of each frame and ultimately locate the interval where the expression occurred. However, the receptive field of CNNs is relatively limited. Each layer of the CNN considers primarily local features, lacking consideration of global features, features between local parts, and features between local and global features. Due to the limited amount of facial expression data currently available for training, overly deep network models are prone to overfitting. Furthermore, because convolutional neural networks insert pooling layers between convolutional layers, facial features, especially subtle features of micro-expressions, can be lost during forward propagation through the network. Therefore, current researchers are using models that are limited in depth. Because the Transformer can detect global image dependencies and transfer features within deep networks, some researchers have combined it with convolutional neural networks for facial expression detection. However, because the Transformer requires a large amount of data for training, and current facial expression databases are relatively small, networks using Transformers often fail to achieve the desired results. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides an expression detection method, system, device and storage medium, which can improve the accuracy of video expression detection.

[0005] In one aspect, an embodiment of the present invention provides a method for detecting facial expressions, comprising the following steps:

[0006] Get the video to be tested;

[0007] Extracting optical flow features of the video to be detected;

[0008] Inputting the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain the first expression detection values ​​of all frames in the video;

[0009] taking a frame corresponding to a first expression detection value greater than a first threshold as a candidate frame;

[0010] Determine a plurality of maximum value points according to the first expression detection values ​​of all the candidate frames;

[0011] The frame with the largest expression amplitude is determined according to the frame distance between the frames corresponding to the two maximum value points, and the frame interval in which the expression exists is determined according to the frame with the largest expression amplitude.

[0012] According to some embodiments of the present invention, the expression detection model includes a first module and multiple second modules, the first module and the multiple second modules are connected in sequence, the first module includes Shifted PatchTokenization and L Swin Transformer Block connected in sequence, the second module includes PatchMerging and L Swin Transformer Block connected in sequence, and the L Swin Transformer Block represents a Swin Transformer module using a local self-attention mechanism.

[0013] According to some embodiments of the present invention, the expression detection model is obtained by the following steps:

[0014] According to the expression interval labels of the training set, determine half of the average number of frames that an expression lasts, denoted as k;

[0015] The OpenCV algorithm is used to calculate the optical flow features between the i-th frame and the i+k-th frame of the video in the training set;

[0016] Select the region of interest in the video based on the preset feature points, and blacken the eye area in the region of interest;

[0017] Redefine the expression window of the training set according to k to obtain a new expression interval label;

[0018] The training set is input into an expression detection model, and the expression detection model is optimized according to a loss function and an optimizer algorithm.

[0019] According to some embodiments of the present invention, the first threshold is obtained by the following steps:

[0020] Determine an expression detection maximum value and an expression detection average value according to the first expression detection values ​​of all frames;

[0021] Subtracting the expression detection average from the expression detection maximum value to obtain an expression detection difference;

[0022] The expression detection difference is multiplied by a preset coefficient and then added to the expression detection average value to obtain a first threshold.

[0023] According to some embodiments of the present invention, inputting the optical flow features of the video to be detected into an expression detection model built based on a SwinTransformer network to obtain first expression detection values ​​for all frames in the video includes the following steps:

[0024] Inputting the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain second expression detection values ​​of all frames;

[0025] The second expression detection value is enhanced to obtain a first expression detection value, wherein the enhancement process is as follows:

[0026]

[0027] Among them, s and Respectively represent the second expression detection value and the first expression detection value of all frames in the video, and s j represents the first expression detection value of the i-th frame and the second expression detection value of the j-th frame, i=F1+k represents that i is the index of the (1+k)-th frame, i=F end -k means i is the index of the kth frame before the end.

[0028] According to some embodiments of the present invention, the L Swin Transformer Block includes a W-MLSA layer, a first MLP layer, a SW-MLSA layer, and a second MLP layer connected in sequence;

[0029] MLP is a multi-layer perceptron, which is used to perform nonlinear classification of input;

[0030] W-MLSA is a multi-head local self-attention mechanism based on a sliding window. The W-MLSA layer is used to move the window in the input feature matrix and calculate the local self-attention of the window.

[0031] SW-MLSA is the offset W-MLSA. The input of the SW-MLSA layer is the output feature matrix of the W-MLSA layer. The SW-MLSA layer is used to calculate the local self-attention within the offset window in the input feature matrix.

[0032] According to some embodiments of the present invention, the Shifted Patch Tokenization is used to shift the optical flow feature matrix by a number of pixels in four directions: upper left, upper right, lower left, and lower right; crop the shifted optical flow feature matrix to the same size as the original optical flow feature matrix and splice it; and split and flatten the spliced ​​matrix to obtain a feature matrix sequence.

[0033] On the other hand, an embodiment of the present invention further provides an expression detection system, comprising:

[0034] The first module is used to obtain the video to be detected;

[0035] The second module is used to extract the optical flow features of the video to be detected;

[0036] The third module is used to input the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain the first expression detection values ​​of all frames in the video;

[0037] A fourth module is configured to select a frame corresponding to a first expression detection value greater than a first threshold as a candidate frame;

[0038] A fifth module is configured to determine a plurality of maximum value points based on the first expression detection values ​​of all the candidate frames;

[0039] The sixth module is used to determine the frame with the largest expression amplitude based on the frame distance between the frames corresponding to the two maximum value points, and determine the frame interval in which the expression exists based on the frame with the largest expression amplitude.

[0040] On the other hand, an embodiment of the present invention further provides an expression detection device, comprising:

[0041] at least one processor;

[0042] at least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor implements the expression detection method as described above.

[0044] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the expression detection method as described above.

[0045] The above technical solution of the present invention has at least one of the following advantages or beneficial effects: extracting the optical flow features of the video to be detected, inputting the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain the first expression detection values ​​of all frames in the video, taking the frames corresponding to the first expression detection values ​​greater than the first threshold as candidate frames, determining multiple maximum points where expressions may exist based on the first expression detection values ​​of all candidate frames, then determining the frame with the largest expression amplitude based on the frame distance between the frames corresponding to the two maximum points, and determining the frame interval where expressions exist based on the frame with the largest expression amplitude. The present application builds an expression detection model based on the Swin Transformer network, which can effectively learn global features. When the expression features are small, the convolutional neural network can also be extended to depth. The expression detection model obtained by training is used to improve the accuracy of the expression detection value, and then the candidate frames are determined based on the obtained expression detection values. The frame interval where expressions exist is determined in combination with the distance between the maximum points of the candidate frames, thereby improving the accuracy of video expression detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the expression detection method provided by an embodiment of the present invention;

[0047] Figure 2 Schematic diagram of the network structure of the expression detection model provided by an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the L Swin Transformer Block structure provided by an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of feature processing in Shifted Patch Tokenization provided by an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the relationship between labels and windows provided by an embodiment of the present invention;

[0051] Figure 6 Schematic diagram of an expression detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar components or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0053] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0054] In the description of the present invention, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or suggesting relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0055] An embodiment of the present invention provides an expression detection method. The expression detection method in the embodiment of the present application can be applied to a terminal or a server, or can be software running on a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0056] Reference Figure 1 The expression detection method of the embodiment of the present invention includes but is not limited to the following steps:

[0057] Step S110, obtaining the video to be detected;

[0058] Step S120, extracting optical flow features of the video to be detected;

[0059] Step S130, inputting the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain first expression detection values ​​of all frames in the video;

[0060] Step S140, taking a frame corresponding to a first expression detection value greater than a first threshold as a candidate frame;

[0061] Step S150, determining a plurality of maximum points according to the first expression detection values ​​of all candidate frames;

[0062] Step S160 , determining the frame with the largest expression amplitude based on the frame distance between the frames corresponding to the two maximum value points, and determining the frame interval in which the expression exists based on the frame with the largest expression amplitude.

[0063] In some embodiments of step S120, optical flow is the instantaneous speed of pixel motion of a spatially moving object on the observation imaging plane. The optical flow method is a method that uses the changes in pixels in the time domain of an image sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. The instantaneous rate of change of grayscale at a specific coordinate point on a two-dimensional image plane is usually defined as an optical flow vector. Optical flow expresses the changes in the image. Since the optical flow feature contains information about the target's motion, it can be used by the observer to determine the target's motion.

[0064] In some embodiments of step S130, the Swin Transformer network can be used as a general backbone model for computer vision problems. The challenge of transferring the Transformer from natural language to vision mainly comes from the differences between the two fields, such as the large differences in the scale of visual entities, the higher resolution of pixels in images compared to words in text, etc. The self-attention calculation in the Swin Transformer is implemented using a shifted window. The shifted window scheme limits the self-attention calculation to non-overlapping local windows while allowing cross-window connections, which brings higher computational efficiency while also enabling connections between features in different local windows. The Swin Transformer adopts a hierarchical architecture, which allows the model to be modeled at different scales, making it adaptable to a wide range of visual tasks, including image classification and dense prediction tasks such as object detection and semantic segmentation. There are many ways to solve optical flow. The optical flow obtained using the TV-L1 method is called TV-L1 optical flow. The first expression detection value output by the expression detection model is used to characterize the probability of the presence of facial expressions in each frame of the image.

[0065] In some embodiments of steps S140 to S150, each frame in the video corresponds to a first expression detection value, and there are multiple maximum points in all the first expression detection values ​​distributed in the time sequence of the frames, and the maximum points that exceed the first threshold are selected for subsequent calculations.

[0066] In some embodiments of step S160, when the distance between the frame corresponding to the selected maximum point and the frame corresponding to other maximum points is greater than k, the maximum point is determined to be The corresponding frame has the largest expression amplitude detected, and the frame interval with expression is Here, k represents half of the average number of frames that an expression lasts in the training set used to train the expression detection model. For the CAS(ME)^2 dataset, the value of k for detecting micro-expressions can be 6, and the value of k for detecting macro-expressions can be 18. It is understandable that the value of k varies depending on the dataset.

[0067] According to some embodiments of the present invention, in step S130, the step of inputting the optical flow features of the video to be detected into the expression detection model built based on the SwinTransformer network to obtain the first expression detection values ​​of all frames in the video includes but is not limited to the following steps:

[0068] Step 210: Inputting the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain second expression detection values ​​of all frames;

[0069] Step 220: Enhance the second expression detection value to obtain the first expression detection value, wherein the enhancement process is as shown in formula (1):

[0070]

[0071] Among them, s and Respectively represent the second expression detection value and the first expression detection value of all frames in the video, and s j represents the first expression detection value of the i-th frame and the second expression detection value of the j-th frame, i=F1+k represents that i is the (Frame) index of the (1+k)-th frame, i=F end -k means i is the index of the kth frame before the end. Formula (1) represents the enhanced s i It is the expression detection value after smoothing the [ik,i+k]th frame interval in the original model output expression detection value, and the enhanced s of each frame i Represents the cumulative confidence score of the presence of expressions within a interval.

[0072] According to some embodiments of the present invention, the first threshold is obtained by the following steps:

[0073] Determine an expression detection maximum value and an expression detection average value according to the first expression detection values ​​of all frames;

[0074] The maximum value of expression detection is subtracted from the average value of expression detection to obtain the expression detection difference;

[0075] The first threshold is obtained by multiplying the expression detection difference by a preset coefficient and adding the resultant to the expression detection average value.

[0076] Specifically, the calculation of the first threshold T is shown in formula (2):

[0077]

[0078] in, represents the average value of expression detection, represents the maximum value of expression detection, and e represents the preset coefficient. The preset coefficient is an adjustable parameter. The value of the preset coefficient can be determined by the performance of the model on a labeled dataset. In the embodiment of the present invention, the preset coefficient can be 0.6.

[0079] According to some embodiments of the present invention, the expression detection model is obtained by the following steps:

[0080] Step S310, determining half of the average number of frames that an expression lasts based on the expression interval labels of the training set, denoted as k;

[0081] Step S320, using the OpenCV algorithm to calculate the optical flow features between the i-th frame and the i+k-th frame of the video in the training set;

[0082] Step S330, selecting a region of interest in the video according to preset feature points, and blackening the eye area in the region of interest;

[0083] Step S340, re-delineating the expression window of the training set according to k to obtain a new expression interval label;

[0084] Step S350: input the training set into the expression detection model, and optimize the expression detection model according to the loss function and the optimizer algorithm.

[0085] Before step S310, the deep neural network face detector of OpenCV can be used to crop the face part of each frame of the video in the training set and re-save it as a (128, 128) size image.

[0086] In some embodiments of step S310, the expression interval label may be to mark frames with expressions in the video as 1, and frames without expressions as 0. The expression interval label may also be [10, 20], indicating that expressions exist from the 10th frame to the 20th frame.

[0087] In some embodiments of step S320, the optical flow features are specifically as follows:

[0088] Assume that the grayscale value of the pixel point (x, y) at time t is I(x, y, t). After time dt, the pixel point moves to (x+dx, y+dy). According to the grayscale conservation law, we can get formula (3):

[0089] I(x,y,t)=I(x+dx,y+dy,t+dt); (3)

[0090] The right side of formula (3) is expanded by Taylor formula to obtain formula (4):

[0091]

[0092] Where β represents the derivative term above the second order. After ignoring the higher-order derivatives, we can get formula (5):

[0093]

[0094] in, and Respectively represent the partial derivatives of the pixel gray value in the image along the three directions of x, y, and t, is the velocity vector of the optical flow in the horizontal direction, is the velocity vector of the optical flow in the vertical direction, and (u, v) is taken as the first two dimensions of the optical flow feature, which is used to solve the optical strain, as shown in formula (6):

[0095]

[0096] Among them, ∈ represents the optical strain, denote the derivatives of u with respect to x and y, respectively. and Denote the derivatives of v with respect to x and y respectively. The amplitude of optical strain |∈| is the third dimension of optical flow characteristics. The amplitude of optical strain can be expressed by the equation Obtain.

[0097] Finally, (u, v, |∈|) as the final optical flow feature can be organized like a normal RGB image and can enter the subsequent processing steps. The dimension of the final optical flow feature is (128, 128, 3).

[0098] In some embodiments of step S330, the dlib toolbox can be used to extract facial feature points. Facial feature points can be regions of interest such as eyes, eyebrows, and mouth. Because the optical flow method used for feature extraction is sensitive to blinking, the eye region will introduce unnecessary noise when extracting optical flow features, which in turn affects subsequent model recognition. Therefore, after extracting the regions of interest, the eye region is filled with black, and then the regions of interest are reassembled into a matrix with dimensions of (42, 42, 3).

[0099] In some embodiments of step S340, a window W is defined based on the average number of frames in which an expression lasts, and labels are re-annotated based on the relationship between the window W and the IoU, such as formulas (7) to (9):

[0100] ε=[F onset ,F offset ]; (7)

[0101]

[0102]

[0103] Combine Figure 5 , F onset Indicates the starting frame of the expression; F apex The vertex frame of the expression, where the expression amplitude reaches its maximum value; F offsett Indicates the end frame of the expression. IoU represents the relationship between the current window W and the original expression interval label ε of the training set. The re-labeled label is obtained based on IoU and function g(IoU).

[0104] In some embodiments of step S350, during the training of the expression detection model, the loss function may use mean square error (MSE), the optimizer algorithm may use stochastic gradient descent (SGD), and the learning rate may be set to 0.0005.

[0105] According to some embodiments of the present invention, referring to Figure 2 The expression detection model based on the Swin Transformer network includes a first module and multiple second modules, which are connected in sequence. The first module includes a Shifted Patch Tokenization and an L Swin Transformer Block connected in sequence, and the second module includes a Patch Merging and an L Swin Transformer Block connected in sequence. The L Swin Transformer Block represents a Swin Transformer module that applies the local self-attention mechanism, namely the LSA (Locality Self-Attention) mechanism. Furthermore, after the last second module, a normalization layer (Normalization) and a multi-layer perceptron (MLP) are connected in sequence, and the expression detection value (score) is finally output.

[0106] Since the amplitude of the expression is small, the corresponding optical flow transformation is small when the expression occurs. Since SwinTransformer does not use pooling layers between layers, the expression optical flow features will not be given the average value or maximum value within the pooling range when being passed forward in the network due to average pooling and max pooling, causing small features to be submerged. Instead, the original features can still be passed forward, so it can be ensured that the features are not lost when being passed forward. The Shifted PatchTokenization and Locality Self-Attention Mechanism used in the improved Swin Transformer architecture can enable the network to achieve performance close to that of a convolutional neural network when the amount of data in the expression database is small. In production and life, when the amount of data is large, the method flow of the present invention can effectively detect expressions.

[0107] According to some embodiments of the present invention, referring to Figure 3 , L Swin Transformer Block includes a W-MLSA layer, a first MLP layer, a SW-MLSA layer, and a second MLP layer connected in sequence. Furthermore, a normalization layer (LN) is connected between the connections of each layer. MLP is a multi-layer perceptron, which is used to perform nonlinear classification of the input. W-MLSA is a multi-head local self-attention mechanism based on a sliding window. The W-MLSA layer is used to move a moving window in the input feature matrix and calculate the local self-attention of the window. SW-MLSA is an offset W-MLSA. The input of the SW-MLSA layer is the output feature matrix of the W-MLSA layer. The SW-MLSA layer is used to calculate the local self-attention within the offset window in the input feature matrix.

[0108] Specifically, the Swin Transformer builds on the basic self-attention mechanism by proposing a sliding window-based self-attention mechanism called Windows Multi-Head Self-Attention (W-MSA). This mechanism uses windows to partition the feature matrix into multiple disjoint regions within each downsampled feature matrix. Self-attention is calculated only within each window, resulting in the final self-attention feature value representing the local features within the current window, compared to the overall feature value. Because the Swin Transformer employs a hierarchical feature map approach similar to that used in convolutional neural networks, the length and width of the feature matrix decrease as it passes through the network due to downsampling. However, for the same window size, the region of the original image that the feature matrix corresponds to gradually increases, representing a transition from considering small-scale local features to large-scale global features. Ultimately, the downsampled feature matrix falls entirely within the window, and the feature matrix calculated within the window no longer represents local features, but rather global features. The aforementioned W-MSA with LSA is now known as W-MLSA.

[0109] Since the Swin Transformer performs self-attention calculations within the window, information cannot be transferred between windows. To solve this problem, the embodiment of the present invention adopts Shifted Windows Multi-Head Self-Attention (SW-MSA), which is a shifted W-MSA. It is used in pairs with W-MSA. When using SW-MSA, the window is shifted from the upper left corner to the right and bottom respectively. pixels, and M is the size of the sliding window W used to calculate self-attention. The input to SW-MSA is the output feature matrix of W-MSA. Therefore, calculating the self-attention within the offset window enables information exchange between windows, thus taking into account the relationship between local features. The SW-MSA applied to LSA is SW-MLSA.

[0110] The embodiment of the present invention adopts a local self-attention mechanism in the process of calculating the self-attention feature, so that the module allocates more attention to other areas when calculating the self-attention. First, the different positions of a single sequence are modeled, and the given sequence is represented by three layers Q, K, and V. The query (Q) and key value (K) are mapped to the output, and the function is as shown in formula (10):

[0111] R=QK T ; (10)

[0112] Among them, Q represents the Q layer, K represents the K layer, and R represents the output of the mapping.

[0113] Then, the elements on the diagonal of the mapped output R are masked so that the self-attention mechanism pays more attention to other areas in the output matrix during operation, rather than just focusing on the area represented by the current Q, as shown in formula (11):

[0114]

[0115] Among them, i, j represent the rows and columns of the matrix respectively, R i, Represents the element in row i and column j of the mapping output above, R M Represents the mapping output after masking.

[0116] Since the self-attention mechanism is used for the window self-attention (Windows Multi-Head Self-Attention (SW-MSA)) and the shifted window self-attention (Shifted Windows Multi-HeadSelf-Attention (SW-MSA)) in the Swin Transformer, the self-attention output of the final window is the mapped output R after masking. M Divided by the learnable parameter τ and added with the offset parameter B, and then normalized by softmax and mapped by V, the result is as shown in formula (12):

[0117]

[0118] Among them, L represents the self-attention matrix finally calculated by Q, K, and V, and R M represents the mapping output after masking, τ represents the learnable range control parameter, B represents the offset parameter, and V represents the V layer, which is used for the final mapping.

[0119] In related art, because the feature distribution dimensions of image information are more extensive and the distribution of attention scores across features is more even, background information also receives more attention, while the attention allocated to important targets is relatively less prominent, meaning that more important features do not receive more attention scores. When the self-attention matrix is ​​calculated based on Q, K, and V when the amount of feature data is small, the most basic self-attention design focuses more on the area where Q is located and less on other areas. The embodiments of the present invention use a masking operation to allow the self-attention calculation to consider more areas in the matrix, rather than just the area represented by the current Q. At the same time, under the influence of the learnable parameter τ and the offset parameter B, the differences in attention distribution between different areas are increased. In general, allowing the self-attention calculation to consider more areas in the matrix and increasing the differences in attention distribution between different areas helps the expression detection model focus on more important areas when training with limited data.

[0120] According to some embodiments of the present invention, Shifted Patch Tokenization is used to shift the optical flow feature matrix by a number of pixels in four directions: upper left, upper right, lower left, and lower right; the shifted optical flow feature matrix is ​​cropped to the same size as the original optical flow feature matrix and spliced; the spliced ​​matrix is ​​split and flattened to obtain a feature matrix sequence.

[0121] Specifically, Shifted Patch Tokenization is used to slice the feature matrix obtained by the optical flow method and input it into the network in a format suitable for the Swin Transformer. The feature processing process in Shifted Patch Tokenization is as follows: Figure 4 As shown:

[0122] Offset the optical flow feature matrix to the upper left, upper right, lower left, and lower right directions pixels, in the embodiment of the present invention, p=7.

[0123] The offset optical flow feature matrix is ​​cropped to the same size as the original optical flow feature matrix and then spliced. The spliced ​​feature is expressed as [x,s 1 ,s 2 ,s 3 ,s 4 ], x represents the original optical flow feature matrix, s 1 ,s 2 ,s 3 ,s 4 They represent the feature matrices after the optical flow feature matrix is ​​offset and cropped in the four directions of upper left, upper right, lower left, and lower right.

[0124] After segmentation, flattening, application layer normalization and linear mapping, the visual marker feature matrix that meets the Swin Transformer format is obtained. The overall process can be expressed by formula (13):

[0125] S(x)=LN(P([x,s 1 ,s 2 ,s 3 ,s 4 ]))E s ; (13)

[0126] Among them, P([x,s 1 ,s 2 ,s 3 ,s 4 ]) means that the input optical flow features and the offset cropped matrix are sliced ​​with size p and flattened into a feature matrix sequence, LN means that the feature matrix obtained in the previous step is normalized by applying layer normalization, E s Represents a linear mapping, and S(x) represents mapping the normalized feature matrix sequence into a visual markup feature matrix suitable for the input of the main architecture of the Swin Transformer network through a fully connected network.

[0127] In the embodiment of the present invention, since Shifted Patch Tokenization is used to replace the embedding layer (PatchEmbedding Layer), the visual mark feature matrix finally input to the Swin Transformer needs to be added with a position offset parameter. The specific process is as shown in formula (14):

[0128] S pe (x)=S(x)+E pos ; (14)

[0129] Among them, S pe (x) represents the visual markup features that are finally input to Swin Transformer, S(x) represents the visual markup features that have been mapped in the previous step, and E pos Represents the position offset parameter.

[0130] The Shifted Patch Tokenization in an embodiment of the present invention adopts an offset operation, which is equivalent to taking into account the spatial relationship between pixels, while also making the receptive field of the model larger. That is, the visual marker feature matrix obtained after linear mapping not only includes the information of the optical flow feature matrix of the original input image, but also includes the information of the pixels near the current area, allowing more spatiotemporal information to be integrated into the training of the main network architecture. Therefore, when the amount of data is small, the network can also obtain relatively sufficient information for training.

[0131] On the other hand, an embodiment of the present invention further provides an expression detection system, comprising:

[0132] The first module is used to obtain the video to be detected;

[0133] The second module is used to extract the optical flow features of the video to be detected;

[0134] The third module is used to input the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain the first expression detection values ​​of all frames in the video;

[0135] A fourth module is configured to select a frame corresponding to a first expression detection value greater than a first threshold as a candidate frame;

[0136] A fifth module is configured to determine a plurality of maximum value points based on the first expression detection values ​​of all the candidate frames;

[0137] The sixth module is used to determine the frame with the largest expression amplitude according to the frame distance between the frames corresponding to the two maximum value points, and determine the frame interval in which the expression exists according to the frame with the largest expression amplitude.

[0138] It can be understood that the contents of the above-mentioned expression detection method embodiments are applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above-mentioned expression detection method embodiments, and the beneficial effects achieved are also the same as those achieved by the above-mentioned expression detection method embodiments.

[0139] Reference Figure 6 , Figure 6 FIG is a schematic diagram of an expression detection device provided by an embodiment of the present invention. The expression detection device of the embodiment of the present invention includes one or more control processors and a memory. Figure 6 A control processor and a memory are taken as an example.

[0140] The control processor and the memory can be connected via a bus or other means. Figure 6 The bus connection is taken as an example.

[0141] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the control processor, and these remote memories may be connected to the expression detection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] Those skilled in the art will understand that Figure 6 The device structure shown in the figure does not constitute a limitation on the expression detection device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0143] The non-transient software program and instructions required to implement the expression detection method applied to the expression detection device in the above embodiment are stored in the memory, and when executed by the control processor, the expression detection method applied to the expression detection device in the above embodiment is executed.

[0144] In addition, an embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by one or more control processors, enabling the one or more control processors to execute the expression detection method in the above method embodiment.

[0145] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0146] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in the relevant technical field without departing from the scope of the present invention.

Claims

1. A method for detecting facial expressions, characterized in that: The following steps are involved: Get the video to be tested; Extracting optical flow features of the video to be detected; Inputting the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain the first expression detection values ​​of all frames in the video; taking a frame corresponding to a first expression detection value greater than a first threshold as a candidate frame; Determine a plurality of maximum value points according to the first expression detection values ​​of all the candidate frames; Determine the frame with the largest expression amplitude based on the frame distance between the frames corresponding to the two maximum value points, and determine the frame interval in which the expression exists based on the frame with the largest expression amplitude; The expression detection model is obtained by the following steps: According to the expression interval labels of the training set, determine half of the average number of frames that an expression lasts, denoted as k; The OpenCV algorithm is used to calculate the optical flow features between the i-th frame and the i+k-th frame of the video in the training set; Select the region of interest in the video based on the preset feature points, and blacken the eye area in the region of interest; Redefine the expression window of the training set according to k to obtain a new expression interval label; Inputting the training set into an expression detection model, and optimizing the expression detection model according to a loss function and an optimizer algorithm; Inputting the optical flow features of the video to be detected into an expression detection model built based on a Swin Transformer network to obtain first expression detection values ​​of all frames in the video includes the following steps: Inputting the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain second expression detection values ​​of all frames; The second expression detection value is enhanced to obtain a first expression detection value, wherein the enhancement process is as follows: Among them, s and Represent the second expression detection value and the first expression detection value of all frames in the video respectively, and S j represents the first expression detection value of the i-th frame and the second expression detection value of the j-th frame, i=F1+k represents that i is the index of the (1+k)-th frame, i=F end -k means i is the index of the kth frame before the end.

2. The facial expression detection method according to claim 1, wherein The expression detection model includes a first module and multiple second modules, the first module and the multiple second modules are connected in sequence, the first module includes Shifted Patch Tokenization and L Swin Transformer Block connected in sequence, the second module includes Patch Merging and L Swin Transformer Block connected in sequence, and the L Swin Transformer Block represents a Swin Transformer module using a local self-attention mechanism.

3. The facial expression detection method according to claim 1, wherein The first threshold is obtained by the following steps: Determine an expression detection maximum value and an expression detection average value according to the first expression detection values ​​of all frames; Subtracting the expression detection average from the expression detection maximum value to obtain an expression detection difference; The expression detection difference is multiplied by a preset coefficient and then added to the expression detection average value to obtain a first threshold.

4. The facial expression detection method according to claim 2, wherein: The L Swin Transformer Block includes a W-MLSA layer, a first MLP layer, a SW-MLSA layer, and a second MLP layer connected in sequence; MLP is a multi-layer perceptron, which is used to perform nonlinear classification of input; W-MLSA is a multi-head local self-attention mechanism based on a sliding window. The W-MLSA layer is used to move the window in the input feature matrix and calculate the local self-attention of the window. SW-MLSA is the offset W-MLSA. The input of the SW-MLSA layer is the output feature matrix of the W-MLSA layer. The W-MLSA layer is used to calculate the local self-attention within the offset window in the input feature matrix.

5. The facial expression detection method according to claim 2, wherein: The Shifted PatchTokenization is used to shift the optical flow feature matrix by a number of pixels in the upper left, upper right, lower left, and lower right directions respectively; crop the shifted optical flow feature matrix to the same size as the original optical flow feature matrix and splice it; split and flatten the spliced ​​matrix to obtain a feature matrix sequence.

6. An expression detection system, characterized in that: include: The first module is used to obtain the video to be detected; The second module is used to extract the optical flow features of the video to be detected; The third module is used to input the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain the first expression detection values ​​of all frames in the video; A fourth module is configured to select a frame corresponding to a first expression detection value greater than a first threshold as a candidate frame; A fifth module is configured to determine a plurality of maximum value points based on the first expression detection values ​​of all the candidate frames; The sixth module is used to determine the frame with the largest expression amplitude based on the frame distance between the frames corresponding to the two maximum value points, and determine the frame interval in which the expression exists based on the frame with the largest expression amplitude; The expression detection model is obtained by the following steps: According to the expression interval labels of the training set, determine half of the average number of frames that an expression lasts, denoted as k; The OpenCV algorithm is used to calculate the optical flow features between the i-th frame and the i+k-th frame of the video in the training set; Select the region of interest in the video based on the preset feature points, and blacken the eye area in the region of interest; Redefine the expression window of the training set according to k to obtain a new expression interval label; Inputting the training set into an expression detection model, and optimizing the expression detection model according to a loss function and an optimizer algorithm; The third module is specifically configured to perform the following steps: Inputting the optical flow features of the video to be detected into an expression detection model built based on the Swin Transformer network to obtain second expression detection values ​​of all frames; The second expression detection value is enhanced to obtain a first expression detection value, wherein the enhancement process is as follows: Among them, s and Represent the second expression detection value and the first expression detection value of all frames in the video respectively, and s j represents the first expression detection value of the i-th frame and the second expression detection value of the j-th frame, i=F1+k represents that i is the index of the (1+k)-th frame, i=F end -k means i is the index of the kth frame before the end.

7. An expression detection device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the expression detection method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to implement the expression detection method according to any one of claims 1 to 5 when executed by the processor.

Images