Video Facial Expression Recognition Method Designed by Combining Sliding Window and Decision Fusion Strategy

Abstract

The present invention provides a video facial expression recognition method designed by combining a sliding window and a decision fusion strategy, including designing a dynamic human face expression recognition network model; obtaining a training set of face images, using the AFEW dataset and the DFEW dataset; preprocessing the training sets in the AFEW dataset and the DFEW dataset; training the network model with the preprocessed datasets; inputting the test sets in the AFEW dataset and the DFEW dataset into the dynamic human face expression recognition model for expression recognition. A decision-level fusion strategy at the global level and the segment level is constructed. Thus, the accuracy of the algorithm model is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video facial expression recognition, and specifically to a video facial expression recognition method designed by combining a sliding window and a decision fusion strategy. Background Art

[0002] In recent years, with the continuous progress of artificial intelligence technology, human-computer interaction has become increasingly important. Facial expression recognition technology has many potential applications in fields such as robotics and computer vision, and has achieved remarkable success. Dynamic facial expression recognition technology can capture the emotional changes of users in real time, enabling machines to respond to users' needs more humanely, thereby greatly improving the user experience and the efficiency of human-computer interaction. Existing dynamic facial expression recognition is based on performing expression recognition on the input video sequence. However, facial expression recognition under natural conditions is closer to the real world. In dynamic facial expression recognition under this background, there are problems such as inter-class similarity and intra-class differences in data samples, occlusion, and data sample imbalance, making dynamic facial expression recognition under natural conditions still challenging.

[0003] In the existing technology, methods based on manually extracting visual features have always been dominant. From the perspective of visual features, there are methods for spatio-temporal feature description such as based on local binary patterns (LBP-TOP, STLBP) and histogram of oriented gradients (HOG-TOP). From the perspective of facial action analysis, action unit coding technologies such as FACS and DISFA are used. These methods make full use of human understanding of expressions and design corresponding feature extraction algorithms. However, these methods also have some deficiencies, such as complex and cumbersome feature design processes, limited feature representation capabilities, the need for a large number of preprocessing steps, and difficulty in implementing end-to-end learning. Summary of the Invention

[0004] The purpose of the present invention is to provide a video facial expression recognition method designed by combining a sliding window and a decision fusion strategy to solve the above-mentioned problems.

[0005] To achieve the above purpose, the present invention provides the following technical solution: A video facial expression recognition method designed by combining a sliding window and a decision fusion strategy, including the following steps:

[0006] Step 1: Design a dynamic facial expression recognition network model, and the network model includes the following parts:

[0007] Sliding sampling window: Input the video into the sliding sampling window module for sampling. The sliding sampling window has a length of w and a height of l = 1. For an input video with a length of L, sampling is performed with a step size of x. The lengths of all video segments are the same, and the frames in the segment are arranged continuously in chronological order. The number of sampled segments n is calculated as follows: Obtain a series of Ci = C1, C2,..., C n video segments of

[0008] Dynamic frame scanning attention: Use four convolutional blocks for intra-segment feature extraction to obtain a four-dimensional feature map F = R NCHW ; where N, C, H, and W represent the number of video frames in the segment, the number of feature channels, the width, and the height of the feature map respectively. For each frame's feature sub-map f = R CHW , use the spatial information extraction module SP (average pooling and max pooling) for extraction operations denoted as f(), calculate the sum of the average and maximum values at the same position (i, j) in all channels, and obtain a two-dimensional matrix m = R HW ; The calculation method is as follows:

[0009]

[0010] In the formula, i ∈ [0, H] and j ∈ [0, W], and k represents the channel; in addition, all the generated spatial feature maps m are concatenated together in chronological order to form a feature block M = R NHW ;

[0011] Obtain the true weight statistics Z ∈ R of the video segment through global convolution instead of conventional global average pooling N ;

[0012]

[0013] W c represents the global convolution kernel of the c-th frame, and Z c represents the true weight value of the c-th frame; send the obtained true weight statistics Z ∈ R N to the self-attention module for learning. First, Z is linearly transformed into query Q, key K, and value V as follows:

[0014]

[0015] where W q , W k , W v ∈ R C×C are trainable weight matrices, and the weight output A is calculated as follows:

[0016]

[0017] T = Norm[MLP(A)] + A

[0018] The scaling factor prevents the vanishing gradient, Norm(·) is the layer normalization operator, and MLP() is the multi-layer perceptron. After re-learning, the adjusted weight T is assigned to the input feature map F = R NCHW , realizing frame-level feature adjustment of the input feature map. The calculation of the adjusted feature map O is as follows:

[0019]

[0020] represents element-wise multiplication by broadcasting; The value range of the output weight T is extended from (0,1) to (0,2). Through the above operations, weights are dynamically assigned, enhancing the weights of effective frames to a certain extent and reducing the impact of noise frames;

[0021] Segment loss function: A segment loss function is designed. For the prediction results of all video segments, the calculation formula is as follows:

[0022]

[0023] The more similar the prediction results of the segments are, the smaller the loss; the prediction results of the segment output are optimized by minimizing this loss function;

[0024] Decision-level fusion: Independent facial expression recognition will be carried out using segment-level features and global-level features respectively, and the recognition results of the two will be fused in the final stage to obtain more accurate and reliable recognition results:

[0025] R gs = R g + R s

[0026] where R g and R s represent the global and segment recognition results respectively, and R gs represents the final recognition result after fusion;

[0027] And the loss function is defined as:

[0028] L = λL Cg + (1 - λ)L Cs + L S

[0029] where λ is a hyperparameter used to balance the two parts, L Cg represents the cross-entropy loss of the global part, L Cs represents the cross-entropy loss of the segment-level part, and L S represents the segment loss;

[0030] Step 2: Obtain the face image training set, using the AFEW dataset and the DFEW dataset;

[0031] Step 3: Preprocess the training sets in the AFEW dataset and the DFEW dataset;

[0032] Step 4: Use the preprocessed dataset to train the network model;

[0033] Step 5: Input the test sets in the AFEW dataset and the DFEW dataset into the dynamic face expression recognition model for expression recognition.

[0034] Preferably, the specific steps of step 1 are as follows:

[0035] Step S11. Through global-level recognition, analyze and understand the entire video, and capture the global features of facial expressions;

[0036] Step S12. Input the video into the sliding sampling window module for sampling. For an input video with a length of L, sample it with a step size of x to obtain a series of C i = C1, C2,..., C n of video segments; where C i represents the i-th segment, and n is the total number of sampled segments;

[0037] Step S13. For each sampled segment, use CNN convolution for feature extraction;

[0038] Step S14. Use dynamic frame scanning attention to evaluate the values of the feature maps and perform frame-level optimization;

[0039] Step S15. Use Transformer to timely perform temporal modeling on the optimized feature maps to capture the subtle expression movements from one frame to another;

[0040] Step S16. Obtain the prediction result of this segment through decision-level fusion of the global-level recognition result and the segment-level recognition result.

[0041] Preferably, the specific steps of step 3 are as follows:

[0042] Step S31. For the AFEW dataset, use Retinaface to detect the face region of the video frame, and then crop and align the face region according to the bounding box and landmarks to 112*112 pixels as the input; and use the skimage library of python to split the video frame by frame;

[0043] Step S32. For the DFEW dataset, all video frames are resized to 112*112 pixels as the input. At the same time, the skimage library in Python is used to frame the video frame by frame. Each video clip gets num pictures. For video clips with less than 16 frames, all frames are directly extracted, and the last frame is copied to supplement 16 pictures. For video clips with num greater than 16, 16 video frames are selected from these pictures. The selection method is: NUMi = i * (num / 16); where NUMi represents the frame number selected from the video frames.

[0044] Step S33. Design an offline data augmentation strategy to perform a series of sample augmentation operations on the original expression dataset, that is, use the mirror flipping technique to flip the original face sequence images to generate a new flipped expression dataset.

[0045] Step S34. Further, perform horizontal mirror flipping on the original dataset and the flipped dataset, and perform random rotation within the range of -30° to 30° to expand the data sample size to 8 times the original.

[0046] Preferably, step S4 specifically includes the following steps:

[0047] Step S41. On the open-source platform of PyTorch, use the GeForce RTX A5000 GPU to train the data sample size. During the training of the dataset, first use DFEW to pre-train the model, and then fine-tune it on the AFEW dataset with the same settings.

[0048] Step S42. Use two metrics, the unweighted average recall rate UAR and the weighted average recall rate WAR, to evaluate the performance of the model. Among them, UAR is calculated by averaging the accuracy of each category without considering the sample size of each category; WAR is the weighted average considering the sample size of each category.

[0049] Preferably, the calculation formula in the step 1 segment loss function: || represents the norm, r i and r j respectively represent the predicted values of the i-th and j-th video segments generated by the temporal module (i ∈ (1, 2, 3,..., n); j ∈ (1, 2, 3,..., n); i ≠ j), n represents the number of segments; in the fractional part of the formula, represents the cosine similarity between two vectors in the range of [-1, 1], and α = 0.3.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] The present invention provides a video facial expression recognition method designed by combining a sliding window and a decision fusion strategy. By constructing a network model with a sliding sampling window and a learning strategy for decision-level fusion of global-level recognition and segment-level recognition, the preprocessed AFEW dataset and DFEW dataset are used for model training, and two metrics, UAR and WAR, are used to evaluate the performance of the model. Moreover, in terms of sampling, the present invention designs a sliding sampling window; in terms of segment-level recognition, the Transformer technology is used to perform temporal modeling on the optimized feature map; in terms of decision-making, a strategy for decision-level fusion of global-level and segment-level is constructed. Thus, the accuracy of the algorithm model is effectively improved. Description of the Drawings

[0052] Figure 1 It is a frame diagram of the face dynamic expression recognition algorithm under natural conditions of the present invention;

[0053] Figure 2 It is the sliding sampling window module of the present invention;

[0054] Figure 3 It is the block diagram of the dynamic frame scanning attention structure of the present invention;

[0055] Figure 4 It is the fragment loss function diagram of the present invention. Detailed Embodiment

[0056] Next, in combination with the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0057] Please refer to the attached Figures 1-4 , the present invention provides a video facial expression recognition method designed by combining a sliding window and a decision fusion strategy, which is characterized by including the following steps:

[0058] Step 1: Design a dynamic face expression recognition network model, which includes the following parts:

[0059] Sliding sampling window: Input the video into the sliding sampling window module for sampling. The sliding sampling window with a length of w and a height of l samples the input video with a length of L at a step size of x to obtain a series of C i = C1, C2,..., C n video segments;

[0060] By evaluating different sliding window sampling settings on the DFEW dataset, this example selects a window with a step size of x = 2, a length of w = 8, and a height of l = 1 for sampling. The specific evaluation is shown in Table 1:

[0061] Table 1: Evaluation of different sliding window sampling settings on DFEW

[0062]

[0063] Dynamic frame scanning attention: Four convolution blocks are used to extract intra-segment features, resulting in a four-dimensional feature map F = R NCHW ; Where N, C, H, and W represent the number of video frames, the number of feature channels, and the width and height of the feature map in the segment, respectively. For each frame’s feature sub-map f=R CHW , use the spatial information extraction module SP (average pooling and maximum pooling) to perform the extraction operation denoted as f(), calculate the average and maximum values of the same position (i, j) in all channels and then sum them up to obtain a two-dimensional matrix m=R containing the spatial information of the frame HW The calculation method is as follows:

[0064]

[0065] In the formula, i∈[0,H] and j∈[0,W], k represents the channel; in addition, all the generated spatial feature maps m are spliced together in time order to form a feature block M=R containing spatiotemporal information NHW ; Obtain the true weight statistics Z∈R of the video clip through global convolution instead of conventional global average pooling N ;

[0066]

[0067] W c Represents the global convolution kernel of the c-th frame, Z c Represents the true weight value of the cth frame; the obtained true weight statistics Z∈R N It is sent to the self-attention module for learning. First, Z linearly converts it into query Q, key K and value V as shown below:

[0068] [Q,K,V]=Z[W q ,W k ,W v ]

[0069] Where W q ,W k ,W v ∈R C×C They are all trainable weight matrices. Note that the weight output A is calculated as follows:

[0070]

[0071] T = Norm[MLP(A)] + A

[0072] The scaling factor prevents the vanishing gradient. Norm(·) is the layer normalization operator, and MLP() is the multi - layer perceptron. After re - learning, the adjusted weight T is assigned to the input feature map F = R NCHW , realizing frame - level feature adjustment of the input feature map. The calculation of the adjusted feature map O is as follows:

[0073]

[0074] represents element - wise multiplication with broadcasting; Thus, the value range of the output weight is extended from (0, 1) to (0, 2). Through the above operations, weights are dynamically allocated, enhancing the weights of valid frames to a certain extent and reducing the impact of noisy frames;

[0075] Segment loss function: A segment loss function is designed. For the prediction results of all video segments, the calculation formula is as follows:

[0076]

[0077] where, || represents the norm, r i and r j respectively represent the predicted values of the i - th and j - th video segments generated by the temporal module (i ∈ (1, 2, 3,..., n); j ∈ (1, 2, 3,..., n); i ≠ j), and n represents the number of segments; in the fractional part of the formula, represents the cosine similarity between two vectors, ranging from [-1, 1], and α = 0.3;

[0078] The more similar the prediction results of the segments are, the smaller the loss; the prediction results of the segment output are optimized by minimizing this loss function;

[0079] Decision - level fusion: Independent facial expression recognition will be carried out using segment - level features and global - level features respectively, and the recognition results of the two will be fused in the final stage to obtain more accurate and reliable recognition results:

[0080] R gs = R g + R s

[0081] where R g and R s respectively represent the global and segment recognition results, and R gsIndicates the final recognition result after fusion;

[0082] And the loss function is defined as:

[0083] L = λL Cg +(1 - λ)L Cs +L S

[0084] Where λ is a hyperparameter used to balance the two parts, L Cg represents the cross-entropy loss of the global part, and L Cs represents the cross-entropy loss of the segment-level part, and L S represents the segment loss; The present invention evaluates different λ on DFEW, and the results are shown in Table 2:

[0085] Table 2: Evaluation of different λ values on DFEW

[0086]

[0087]

[0088] Step 2: Obtain the face image training set, using the AFEW dataset and the DFEW dataset;

[0089] Step 3: Preprocess the training sets in the AFEW dataset and the DFEW dataset; Specifically, it includes the following steps:

[0090] Step S31. For the AFEW dataset, use Retinaface to detect the face area of the video frame, and then crop and align the face area according to the bounding box and landmarks to 112*112 pixels as the input; And use the skimage library of python to frame the video frame by frame;

[0091] Step S32. For the DFEW dataset, all video frames are adjusted to 112*112 pixels as the input; At the same time, use the skimage library of python to frame the video frame by frame. Each video segment gets num pictures. For video segments with less than 16 frames, directly extract all frames and copy the last frame to supplement 16 pictures. For video segments with num greater than 16, select 16 video frames from these pictures. The selection method is: NUMi = i*(num / 16); Where NUMi represents the frame number selected from the video frame;

[0092] Step S33. Design an offline data augmentation strategy to perform a series of sample augmentation operations on the original expression dataset, that is, use the mirror flipping technology to flip the original face sequence images, so as to generate a new flipped expression dataset;

[0093] Step S34. Further perform horizontal mirror flipping on the original dataset and the flipped dataset, and perform random rotation within the range of -30° to 30°, so that the data sample size is expanded to 8 times the original;

[0094] Step 4: Use the preprocessed dataset to train the network model; specifically including the following steps:

[0095] Step S41. On the open-source platform of PyTorch, use the GeForce RTX A5000 GPU to train the data sample size; during the training process of the dataset, first use DFEW to pre-train the model, and then perform fine-tuning on the AFEW dataset with the same settings; the case environment parameters are set as shown in Table 3:

[0096] Table 3: Environmental parameter settings

[0097]

[0098] Step S42. Adopt two metrics, the unweighted average recall rate UAR and the weighted average recall rate WAR, to evaluate the performance of the model, where UAR is obtained by averaging the accuracy of each category, without considering the sample quantity of each category; WAR is the weighted average considering the sample quantity of each category; the evaluation of different fusion strategies in DFEW is shown in Table 4, and the evaluation of each component of the present invention is shown in Table 5:

[0099] Table 4: Evaluation of different fusion strategies in DFEW

[0100]

[0101] Table 5: Evaluation of each component of the present invention

[0102]

[0103]

[0104] Step 5: Input the test sets in the AFEW dataset and the DFEW dataset into the dynamic facial expression recognition model for facial expression recognition. After the recognition is completed, compare with the state-of-the-art method on DFEW, and the results are shown in Table 6:

[0105] Table 6: Comparison results between the present technology and the state-of-the-art method on DFEW

[0106]

[0107]

[0108] In summary, through the video facial expression recognition method designed by combining the sliding window and decision fusion strategy recorded in the present invention, the UAR and WAR results of the present invention are improved compared with the prior art, thus effectively improving the accuracy of the algorithm model.

[0109] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A video facial expression recognition method designed by combining a sliding window and a decision fusion strategy, characterized in that It includes the following steps: Step 1: Design a dynamic facial expression recognition network model, which includes the following parts: Sliding sampling window: The video is input into the sliding sampling window module for sampling. The sliding sampling window has a length of w and a height of l = 1, and samples the input video with a length of L at a step size of x; all video segments have the same length, and the frames in the segment are arranged continuously in chronological order; the number of sampled segments n is calculated as follows: A series of C i = C1, C2,..., C n video segments; Dynamic Frame Scanning Attention: Use four convolutional blocks for intra-segment feature extraction to obtain a four-dimensional feature map F = R NCHW ; where N, C, H, and W represent the number of video frames, the number of feature channels, the width, and the height of the feature map within the segment, respectively; for each frame's feature sub-map f = R CHW , use the spatial information extraction module SP (average pooling and max pooling) for extraction operations denoted as f(), calculate the sum of the average and maximum values at the same position (i, j) in all channels, and obtain a two-dimensional matrix m = R containing the spatial information of the frame HW ; The calculation method is as follows: In the formula, \(i\in[0, H]\) and \(j\in[0, W]\), where \(k\) represents the channel; in addition, all the generated spatial feature maps \(m\) are concatenated together in chronological order to form a feature block \(M = R\) containing spatio-temporal information NHW ; Obtain the true weight statistics Z ∈ R of the video clip through global convolution instead of conventional global average pooling N ; W c represents the global convolution kernel of the c-th frame, Z c represents the true weight value of the c-th frame; the obtained true weight statistics Z ∈ R N are sent to the self-attention module for learning. First, Z linearly transforms it into query Q, key K, and value V as follows: [Q, K, V] = Z[W q , W k , W v ​ Among which W q , W k , W v ∈R C×C is a trainable weight matrix, and the weight output A is calculated as follows: T = Norm[MLP(A)] + A The scaling factor prevents gradient vanishing, Norm(·) is the layer normalization operator, and MLP() is the multi-layer perceptron. After re-learning, the adjusted weight T is assigned to the input feature map F = R NCHW , to achieve frame-level feature adjustment of the input feature map. The calculation of the adjusted feature map O is as follows: Indicates broadcast element multiplication; Expand the value range of the output weight T from (0, 1) to (0, 2). Through the above operations, the weights are dynamically allocated, enhancing the weights of valid frames to a certain extent and reducing the impact of noise frames; Segment loss function: A segment loss function is designed. For the prediction results of all video segments, the calculation formula is as follows: The more similar the prediction results for the segments are, the smaller the loss; the prediction results of the segment output are optimized by minimizing this loss function; Decision-level fusion: Independent facial expression recognition is carried out using segment-level features and global-level features respectively, and the recognition results of the two are fused in the final stage to obtain more accurate and reliable recognition results: R gs = R g + R s where R g and R s represent the global and segment recognition results respectively, and R gs represents the final recognition result after fusion; And the loss function is defined as: L = λL Cg +(1 - λ)L Cs +L S where λ is a hyperparameter used to balance the two parts, L Cg represents the cross-entropy loss of the global part, L Cs represents the cross-entropy loss of the segment-level part, L S represents the segment loss; Step 2: Obtain a face image training set, using the AFEW dataset and the DFEW dataset; Step 3: Preprocess the training sets in the AFEW dataset and the DFEW dataset; Step 4: Use the preprocessed dataset to train the network model; Step 5: Input the test sets in the AFEW dataset and the DFEW dataset into the dynamic facial expression recognition model for expression recognition.

2. The video facial expression recognition method designed by combining a sliding window and a decision fusion strategy according to claim 1, characterized in that, The specific steps of Step 1 include the following steps: Step S11. Through global-level recognition, analyze and understand the entire video to capture the global features of facial expressions; Step S12. Input the video into the sliding sampling window module for sampling. Sample the input video with length L at a step size of x to obtain a series of video segments C i = C1, C2,..., C n ; where C i represents the i-th segment, and n is the total number of sampled segments; Step S13. For each sampled segment, use CNN convolution for feature extraction; Step S14. Use dynamic frame scanning attention to evaluate the values of the feature maps and perform frame-level optimization; Step S15. Use Transformer to perform temporal modeling on the optimized feature maps in a timely manner to capture the subtle expression movements from one frame to another; Step S16. Obtain the prediction result of this segment through decision-level fusion of the global-level recognition result and the segment-level recognition result.

3. The video facial expression recognition method designed by combining a sliding window and a decision fusion strategy according to claim 1, characterized in that, The specific steps of Step 3 include the following steps: Step S31. For the AFEW dataset, use Retinaface to detect the face area of the video frame, and then crop and align the face area according to the bounding box and landmarks to 112*112 pixels as the input; and use the skimage library of python to frame the video frame by frame; Step S32. For the DFEW dataset, all video frames are adjusted to 112*112 pixels as the input; at the same time, use the skimage library of python to frame the video frame by frame. Each video segment gets num pictures. For video segments with less than 16 frames, directly extract all frames and copy the last frame to supplement 16 pictures. For video segments with num greater than 16, select 16 video frames from these pictures. The selection method is: NUMi = i*(num / 16); where NUMi represents the frame number sequence selected from the video frames; Step S33. Design an offline data augmentation strategy to perform a series of sample augmentation operations on the original expression dataset, that is, use the mirror flipping technology to flip the original face sequence images to generate a new flipped expression dataset; Step S34. Further perform horizontal mirror flipping on the original dataset and the flipped dataset, and randomly rotate them within the range of -30° to 30° to expand the data sample size to 8 times the original.

4. The video facial expression recognition method designed by combining a sliding window and a decision fusion strategy according to claim 1, characterized in that, Step S4 specifically includes the following steps: Step S41. On the open-source platform of PyTorch, use the GeForce RTX A5000 GPU to train the data sample size; during the training of the dataset, first use DFEW to pre-train the model, and then fine-tune it with the same settings on the AFEW dataset; Step S42. Use two metrics, the unweighted average recall rate UAR and the weighted average recall rate WAR, to evaluate the performance of the model, where UAR is obtained by averaging the accuracy of each category without considering the number of samples in each category; WAR is the weighted average considering the number of samples in each category.

5. The video facial expression recognition method designed by combining a sliding window and a decision fusion strategy according to claim 1, characterized in that, The calculation formula in the step 1 segment loss function: || represents the norm, r i and r j respectively represent the predicted values of the i-th and j-th video segments generated by the temporal module (i ∈ (1, 2, 3,..., n); j ∈ (1, 2, 3,..., n); i ≠ j), where n represents the number of segments; in the fractional part of the formula, represents the cosine similarity between two vectors in the range of [-1, 1], and α = 0.3.

Images