Video facial expression recognition method combining sliding window and decision fusion strategy design

Abstract

The invention provides a video facial expression recognition method designed by combining a sliding window and a decision fusion strategy. The method comprises the following steps: designing a dynamic facial expression recognition network model; acquiring a face image training set, and adopting an AFEW data set and a DFEW data set; training sets in the AFEW data set and the DFEW data set are preprocessed; training a network model by using the preprocessed data set; and inputting the test sets in the AFEW data set and the DFEW data set into the dynamic facial expression recognition model, and performing expression recognition. And a strategy of decision level fusion of a global level and a fragment level is constructed. Therefore, the accuracy of the algorithm model is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video facial expression recognition, in particular to a video facial expression recognition method designed by combining a sliding window and a decision fusion strategy. Background Art

[0002] In recent years, with the continuous advancement of artificial intelligence technology, human-computer interaction has become increasingly important. Facial expression recognition technology has many potential applications in fields such as robotics and computer vision, and has achieved remarkable success. Dynamic facial expression recognition technology can capture the user's emotional changes in real time, enabling the machine to respond to the user's needs in a more humane way, thereby greatly improving the user experience and the efficiency of human-computer interaction. Existing dynamic facial expression recognition is based on expression recognition of input video sequences, but facial expression recognition under natural conditions is closer to the real world. In dynamic facial expression recognition under this background, there are problems such as similarity between data samples and differences within classes, occlusion, and imbalance of data samples, which makes dynamic facial expression recognition under natural conditions still challenging.

[0003] Among the existing technologies, methods based on manually extracted visual features have always dominated. From the perspective of visual features, there are methods based on spatiotemporal feature descriptions such as local binary patterns (LBP-TOP, STLBP) and histograms of oriented gradients (HOG-TOP). From the perspective of facial action analysis, action unit encoding technologies such as FACS and DISFA are used. These methods make full use of human understanding of facial expressions and design corresponding feature extraction algorithms. However, these methods also have some shortcomings, such as the complex and cumbersome feature design process, limited feature representation capabilities, the need for a large number of preprocessing steps, and difficulty in achieving end-to-end learning. Summary of the invention

[0004] The purpose of the present invention is to provide a video facial expression recognition method combining sliding window and decision fusion strategy design to solve the above-mentioned problems.

[0005] To achieve the above object, the present invention provides the following technical solution: a video facial expression recognition method designed in combination with a sliding window and a decision fusion strategy, comprising the following steps:

[0006] Step 1: Design a dynamic facial expression recognition network model. The network model includes the following parts:

[0007] Sliding sampling window: The video is input to the sliding sampling window module for sampling. The sliding sampling window with length w and height l=1 samples the input video with a step size of x for the length L. All video segments have the same length, and the frames in the segments are arranged continuously in chronological order. The number of sampled segments n is calculated as follows: Get a series of Ci =C1,C2,...,C n Video clips of .

[0008] Dynamic frame scanning attention: Use four convolutional blocks to extract intra-fragment features and obtain a four-dimensional feature map F = R NCHW ; Where N, C, H, and W represent the number of video frames, the number of feature channels, and the width and height of the feature map in the clip, respectively. For each frame’s feature sub-map f = R CHW , use the spatial information extraction module SP (average pooling and maximum pooling) to perform the extraction operation, denoted as f(), calculate the average and maximum values ​​of the same position (i, j) in all channels and then sum them up to obtain a two-dimensional matrix m=R containing the spatial information of the frame HW ; The calculation method is as follows:

[0009]

[0010] In the formula, i∈[0,H] and j∈[0,W], k represents the channel; in addition, all the generated spatial feature maps m are spliced ​​together in time order to form a feature block M=R containing spatiotemporal information NHW ;

[0011] The true weight statistics Z∈R of the video clip are obtained by global convolution instead of conventional global average pooling N ;

[0012]

[0013] W c represents the global convolution kernel of the cth frame, Z c Represents the true weight value of the cth frame; the true weight statistics Z∈R N Sent to the self-attention module for learning, first Z linearly transforms it into query Q, key K and value V, as shown below:

[0014]

[0015] Where W q ,W k ,W v ∈R C×C is a trainable weight matrix, and the weight output A is calculated as follows:

[0016]

[0017] T=Norm[MLP(A)]+A

[0018] is the scaling factor to prevent the gradient from disappearing, Norm(·) is the layer normalization operator, MLP() is the multi-layer perceptron, and after relearning, the adjusted weight T is assigned to the input feature map F = R NCHW , realize the frame-level feature adjustment of the input feature map, and the calculation of the adjusted feature map O is as follows:

[0019]

[0020] represents broadcast element-wise multiplication; The value range of the output weight T is expanded from (0,1) to (0,2). Through the above operation, the weight is dynamically allocated, which can enhance the weight of the valid frame to a certain extent and reduce the impact of the noise frame.

[0021] Clip loss function: A clip loss function is designed. The calculation formula for the prediction results of all video clips is as follows:

[0022]

[0023] The more similar the prediction results for the segments are, the smaller the loss is; the prediction results of the segment output are optimized by minimizing the loss function;

[0024] Decision-level fusion: segment-level features and global-level features are used to perform independent facial expression recognition, and the recognition results of the two are fused in the final stage to obtain more accurate and reliable recognition results:

[0025] R gs =R g +R s

[0026] Where R g and R s Represent the global and fragment recognition results respectively, R gs Represents the final recognition result after fusion;

[0027] And the loss function is defined as:

[0028] L=λL Cg +(1-λ)L Cs +L S

[0029] Among them, λ is a hyperparameter used to balance the two parts, L Cg represents the cross entropy loss of the global part, L Cs represents the cross entropy loss of the segment level part, L S Indicates fragment loss;

[0030] Step 2: Obtain a face image training set, using the AFEW dataset and the DFEW dataset;

[0031] Step 3: Preprocess the training sets in the AFEW dataset and the DFEW dataset;

[0032] Step 4: Use the preprocessed data set to train the network model;

[0033] Step 5: Input the test sets in the AFEW dataset and the DFEW dataset into the dynamic facial expression recognition model for expression recognition.

[0034] Preferably, the step 1 specifically comprises the following steps:

[0035] Step S11. Analyze and understand the entire video through global level recognition to capture the global features of facial expressions;

[0036] Step S12: Input the video to the sliding sampling window module for sampling. For the input video with a length of L, sample it with a step length of x to obtain a series of C i =C1,C2,...,C n video clip; among them C i represents the i-th fragment, and n is the total number of sampled fragments;

[0037] Step S13. For each sampled segment, use CNN convolution to perform feature extraction;

[0038] Step S14. Use dynamic frame scanning attention to evaluate the value of the feature map and perform frame-level optimization;

[0039] Step S15. Use Transformer to model the optimized feature map in time to capture subtle expression movements from one frame to another;

[0040] Step S16: Obtain the prediction result of the segment by performing decision-level fusion of the global-level recognition result and the segment-level recognition result.

[0041] Preferably, the step 3 specifically comprises the following steps:

[0042] Step S31. For the AFEW dataset, use Retinaface face detection to detect the face area of ​​the video frame, then crop and align the face area to 112*112 pixels according to the bounding box and landmarks as input; and use the skimage library of Python to divide the video frame by frame;

[0043] Step S32. For the DFEW dataset, all video frames are adjusted to 112*112 pixels as input; at the same time, the skimage library of Python is used to divide the video frame by frame, and num pictures are obtained for each video segment. For video segments with less than 16 frames, all frames are directly extracted, and the last frame is copied to supplement 16 pictures. For video segments with num greater than 16, 16 video frames are selected from these pictures, and the selection method is: NUMi=i*(num / 16); where NUMi represents the frame number selected from the video frame;

[0044] Step S33. Design an offline data enhancement strategy to perform a series of sample expansion operations on the original expression data set, that is, use the mirror flip technology to flip the original face sequence image, thereby generating a new flipped expression data set;

[0045] Step S34. The original data set and the flipped data set are further horizontally mirror-flipped and randomly rotated within the range of -30° to 30°, so that the data sample size is expanded to 8 times the original amount.

[0046] Preferably, step S4 specifically includes the following steps:

[0047] Step S41. On the open source platform PyTorch, the data sample size is trained using the GeForce RTX A5000 GPU; during the training of the data set, the model is first pre-trained using DFEW, and then fine-tuned on the AFEW data set with the same settings;

[0048] Step S42. The unweighted average recall rate (UAR) and the weighted average recall rate (WAR) are used to evaluate the performance of the model. UAR is calculated by averaging the accuracy of each category without considering the number of samples in each category; WAR is a weighted average of the number of samples in each category.

[0049] Preferably, the calculation formula in the segment loss function of step 1 is: || represents the norm, r i and r j Respectively represent the predicted values ​​of the i-th and j-th video segments generated by the timing module (i∈(1,2,3,...,n); j∈(1,2,3,...,n); i≠j), n represents the number of segments; the fraction part of the formula It means that the cosine similarity between two vectors is in the range of [-1,1], and α=0.3.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention provides a video facial expression recognition method designed by combining sliding window and decision fusion strategy. By constructing a network model with sliding sampling window, and constructing a learning strategy of decision-level fusion of global-level recognition and segment-level recognition, the pre-processed AFEW data set and DFEW data set are trained, and the performance of the model is evaluated by using two indicators, UAR and WAR. In terms of sampling, the present invention designs a sliding sampling window; in terms of segment-level recognition, the Transformer technology is used to model the optimized feature map in time; in terms of decision-making, a strategy of decision-level fusion of global level and segment level is constructed. Thus, the accuracy of the algorithm model is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a framework diagram of the algorithm for recognizing dynamic facial expressions under natural conditions of the present invention;

[0053] Figure 2 It is a sliding sampling window module of the present invention;

[0054] Figure 3 This is a block diagram of the dynamic frame scanning attention structure of the present invention;

[0055] Figure 4 This is a fragment loss function diagram of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] Please refer to the attached Figure 1-4 The present invention provides a video facial expression recognition method combining sliding window and decision fusion strategy design, characterized in that it includes the following steps:

[0058] Step 1: Design a dynamic facial expression recognition network model. The network model includes the following parts:

[0059] Sliding sampling window: The video is input into the sliding sampling window module for sampling. The sliding sampling window with a length of w and a height of l samples the input video with a step size of x for a length of L, and obtains a series of C i =C1,C2,...,C n Video clips of;

[0060] By evaluating different sliding window sampling settings on the DFEW dataset, this example selects a window with a step size of x=2, a length of w=8, and a height of l=1 for sampling. The specific evaluation is shown in Table 1:

[0061] Table 1: Evaluation of different sliding window sampling settings on DFEW

[0062]

[0063] Dynamic frame scanning attention: Use four convolutional blocks to extract intra-fragment features and obtain a four-dimensional feature map F = R NCHW ; Where N, C, H, and W represent the number of video frames, the number of feature channels, and the width and height of the feature map in the clip, respectively. For each frame’s feature sub-map f = R CHW , use the spatial information extraction module SP (average pooling and maximum pooling) to perform the extraction operation, denoted as f(), calculate the average and maximum values ​​of the same position (i, j) in all channels and then sum them up to obtain a two-dimensional matrix m=R containing the spatial information of the frame HW ; The calculation method is as follows:

[0064]

[0065] In the formula, i∈[0,H] and j∈[0,W], k represents the channel; in addition, all the generated spatial feature maps m are spliced ​​together in time order to form a feature block M=R containing spatiotemporal information NHW ; The true weight statistics Z∈R of the video clip are obtained by global convolution instead of conventional global average pooling N ;

[0066]

[0067] W c represents the global convolution kernel of the cth frame, Z c Represents the true weight value of the cth frame; the true weight statistics Z∈R N Sent to the self-attention module for learning, first Z linearly transforms it into query Q, key K and value V, as shown below:

[0068] [Q,K,V]=Z[W q ,W k ,W v ]

[0069] Where W q ,W k ,W v ∈R C×C are all trainable weight matrices. Note that the weight output A is calculated as follows:

[0070]

[0071] T=Norm[MLP(A)]+A

[0072] is the scaling factor to prevent the gradient from disappearing, Norm(·) is the layer normalization operator, MLP() is the multi-layer perceptron, and after relearning, the adjusted weight T is assigned to the input feature map F = R NCHW , realize the frame-level feature adjustment of the input feature map, and the calculation of the adjusted feature map O is as follows:

[0073]

[0074] represents broadcast element-wise multiplication; Thus, the value range of the output weight is expanded from (0,1) to (0,2). Through the above operation, the weight is dynamically allocated, which can enhance the weight of the valid frame to a certain extent and reduce the influence of the noise frame.

[0075] Clip loss function: A clip loss function is designed. The calculation formula for the prediction results of all video clips is as follows:

[0076]

[0077] Among them, || represents the norm, r i and r j Respectively represent the predicted values ​​of the i-th and j-th video segments generated by the timing module (i∈(1,2,3,...,n); j∈(1,2,3,...,n); i≠j), n represents the number of segments; the fraction part of the formula Indicates that the cosine similarity between two vectors is in the range of [-1,1], and α = 0.3;

[0078] The more similar the prediction results for the segments are, the smaller the loss is; the prediction results of the segment output are optimized by minimizing the loss function;

[0079] Decision-level fusion: segment-level features and global-level features are used to perform independent facial expression recognition, and the recognition results of the two are fused in the final stage to obtain more accurate and reliable recognition results:

[0080] R gs =R g +R s

[0081] Where R g and R s Represent the global and fragment recognition results respectively, R gsRepresents the final recognition result after fusion;

[0082] And the loss function is defined as:

[0083] L=λL Cg +(1-λ)L Cs +L S

[0084] Among them, λ is a hyperparameter used to balance the two parts, L Cg represents the cross entropy loss of the global part, L Cs represents the cross entropy loss of the segment level part, L S Indicates the fragment loss; the present invention evaluates different λ on DFEW, and the results are shown in Table 2:

[0085] Table 2: Evaluation of different λ values ​​on DFEW

[0086]

[0087]

[0088] Step 2: Obtain a face image training set, using the AFEW dataset and the DFEW dataset;

[0089] Step 3: Preprocess the training sets in the AFEW dataset and the DFEW dataset; specifically, the following steps are included:

[0090] Step S31. For the AFEW dataset, use Retinaface face detection to detect the face area of ​​the video frame, then crop and align the face area to 112*112 pixels according to the bounding box and landmarks as input; and use the skimage library of Python to divide the video frame by frame;

[0091] Step S32. For the DFEW dataset, all video frames are adjusted to 112*112 pixels as input; at the same time, the skimage library of Python is used to divide the video frame by frame, and num pictures are obtained for each video segment. For video segments with less than 16 frames, all frames are directly extracted, and the last frame is copied to supplement 16 pictures. For video segments with num greater than 16, 16 video frames are selected from these pictures, and the selection method is: NUMi=i*(num / 16); where NUMi represents the frame number selected from the video frame;

[0092] Step S33. Design an offline data enhancement strategy to perform a series of sample expansion operations on the original expression data set, that is, use the mirror flip technology to flip the original face sequence image, thereby generating a new flipped expression data set;

[0093] Step S34. Further horizontally mirror-flip the original data set and the flipped data set, and randomly rotate them within the range of -30° to 30°, so that the data sample size is expanded to 8 times of the original;

[0094] Step 4: Use the preprocessed data set to train the network model; specifically, the following steps are included:

[0095] Step S41. On the open source platform PyTorch, the data sample size is trained using the GeForce RTX A5000 GPU. During the training of the data set, the model is first pre-trained using DFEW, and then fine-tuned on the AFEW data set with the same settings. The case environment parameter settings are shown in Table 3:

[0096] Table 3: Environmental parameter settings

[0097]

[0098] Step S42. The performance of the model is evaluated by using two indicators, the unweighted average recall rate UAR and the weighted average recall rate WAR, where UAR is calculated by averaging the accuracy of each category without considering the number of samples in each category; WAR is the weighted average of the number of samples in each category; the evaluation of different fusion strategies in DFEW is shown in Table 4, and the evaluation of each component of the present invention is shown in Table 5:

[0099] Table 4: Evaluation of different fusion strategies in DFEW

[0100]

[0101] Table 5: Evaluation of each component of the present invention

[0102]

[0103]

[0104] Step 5: Input the test set in the AFEW dataset and the DFEW dataset into the dynamic facial expression recognition model for expression recognition. After recognition, it is compared with the most advanced method on DFEW. The results are shown in Table 6:

[0105] Table 6: Comparison results between our technique and the most advanced methods on DFEW

[0106]

[0107]

[0108] In summary, through the video facial expression recognition method combined with the sliding window and decision fusion strategy design described in the present invention, the UAR and WAR results of the present invention are improved compared with the prior art, thereby effectively improving the accuracy of the algorithm model.

[0109] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A video facial expression recognition method designed by combining sliding window and decision fusion strategy, characterized in that: The following steps are involved: Step 1: Design a dynamic facial expression recognition network model. The network model includes the following parts: Sliding sampling window: The video is input to the sliding sampling window module for sampling. The sliding sampling window with a length of w and a height of l=1 samples the input video with a length of L and a step size of x. All video segments have the same length, and the frames in the segments are arranged continuously in chronological order. The number of sampled segments n is calculated as follows: Get a series of C i =C1,C2,...,C n Video clips of . Dynamic frame scanning attention: Use four convolutional blocks to extract intra-fragment features and obtain a four-dimensional feature map F = R NCHW ; Where N, C, H, and W represent the number of video frames, the number of feature channels, and the width and height of the feature map in the clip, respectively. For each frame’s feature sub-map f = R CHW , use the spatial information extraction module SP (average pooling and maximum pooling) to perform the extraction operation, denoted as f(), calculate the average and maximum values ​​of the same position (i, j) in all channels and then sum them up to obtain a two-dimensional matrix m=R containing the spatial information of the frame HW ; The calculation method is as follows: In the formula, i∈[0,H] and j∈[0,W], k represents the channel; in addition, all the generated spatial feature maps m are spliced ​​together in time order to form a feature block M=R containing spatiotemporal information NHW ; The true weight statistics Z∈R of the video clip are obtained by global convolution instead of conventional global average pooling. N ; W c represents the global convolution kernel of the cth frame, Z c Represents the true weight value of the cth frame; the obtained true weight statistics Z∈R N Sent to the self-attention module for learning, first Z linearly transforms it into query Q, key K and value V, as shown below: [Q,K,V]=Z[W q ,W k ,W v ] Where W q ,W k ,W v ∈R C×C is a trainable weight matrix, and the weight output A is calculated as follows: T=Norm[MLP(A)]+A is the scaling factor to prevent the gradient from disappearing, Norm(·) is the layer normalization operator, MLP() is the multi-layer perceptron, and after relearning, the adjusted weight T is assigned to the input feature map F = R NCHW , realize the frame-level feature adjustment of the input feature map, and the calculation of the adjusted feature map O is as follows: represents broadcast element-wise multiplication; The value range of the output weight T is expanded from (0,1) to (0,2). Through the above operation, the weight is dynamically allocated, which can enhance the weight of the valid frame to a certain extent and reduce the impact of the noise frame. Clip loss function: A clip loss function is designed. The calculation formula for the prediction results of all video clips is as follows: The more similar the prediction results for the segments are, the smaller the loss is; the prediction results of the segment output are optimized by minimizing the loss function; Decision-level fusion: segment-level features and global-level features are used to perform independent facial expression recognition, and the recognition results of the two are fused in the final stage to obtain more accurate and reliable recognition results: R gs =R g +R s Where R g and R s Represent the global and fragment recognition results respectively, R gs Represents the final recognition result after fusion; And the loss function is defined as: L=λL Cg +(1-λ)L Cs +L S Among them, λ is a hyperparameter used to balance the two parts, L Cg represents the cross entropy loss of the global part, L Cs represents the cross entropy loss of the segment level part, L S Indicates fragment loss; Step 2: Obtain a face image training set, using the AFEW dataset and the DFEW dataset; Step 3: Preprocess the training sets in the AFEW dataset and the DFEW dataset; Step 4: Use the preprocessed data set to train the network model; Step 5: Input the test sets in the AFEW dataset and the DFEW dataset into the dynamic facial expression recognition model for expression recognition.

2. The video facial expression recognition method according to claim 1, characterized in that: The step 1 specifically comprises the following steps: Step S11. Analyze and understand the entire video through global level recognition to capture the global features of facial expressions; Step S12: Input the video to the sliding sampling window module for sampling. For the input video with a length of L, sample it with a step length of x to obtain a series of C i =C1,C2,...,C n video clip; among them C i represents the i-th fragment, and n is the total number of sampled fragments; Step S13. For each sampled segment, use CNN convolution to perform feature extraction; Step S14. Use dynamic frame scanning attention to evaluate the value of the feature map and perform frame-level optimization; Step S15. Use Transformer to model the optimized feature map in time to capture subtle expression movements from one frame to another; Step S16: Obtain the prediction result of the segment by performing decision-level fusion of the global-level recognition result and the segment-level recognition result.

3. The video facial expression recognition method according to claim 1, characterized in that: The step 3 specifically comprises the following steps: Step S31. For the AFEW dataset, use Retinaface face detection to detect the face area of ​​the video frame, then crop and align the face area to 112*112 pixels according to the bounding box and landmarks as input; and use the skimage library of Python to divide the video frame by frame; Step S32. For the DFEW dataset, all video frames are adjusted to 112*112 pixels as input; at the same time, the skimage library of Python is used to divide the video frame by frame, and num pictures are obtained for each video segment. For video segments with less than 16 frames, all frames are directly extracted, and the last frame is copied to supplement 16 pictures. For video segments with num greater than 16, 16 video frames are selected from these pictures, and the selection method is: NUMi=i*(num / 16); where NUMi represents the frame number selected from the video frame; Step S33. Design an offline data enhancement strategy to perform a series of sample expansion operations on the original expression data set, that is, use the mirror flip technology to flip the original face sequence image, thereby generating a new flipped expression data set; Step S34. The original data set and the flipped data set are further horizontally mirror-flipped and randomly rotated within the range of -30° to 30°, so that the data sample size is expanded to 8 times the original amount.

4. The video facial expression recognition method according to claim 1, characterized in that: Step S4 specifically includes the following steps: Step S41. On the open source platform PyTorch, the data sample size is trained using the GeForce RTX A5000 GPU; during the training of the data set, the model is first pre-trained using DFEW, and then fine-tuned on the AFEW data set with the same settings; Step S42. The unweighted average recall rate (UAR) and the weighted average recall rate (WAR) are used to evaluate the performance of the model. UAR is calculated by averaging the accuracy of each category without considering the number of samples in each category; WAR is a weighted average of the number of samples in each category.

5. The video facial expression recognition method according to claim 1, characterized in that: The calculation formula in the fragment loss function of step 1 is: || represents the norm, r i and r j Respectively represent the predicted values ​​of the i-th and j-th video segments generated by the timing module (i∈(1,2,3,...,n); j∈(1,2,3,...,n); i≠j), n represents the number of segments; the fraction part of the formula It means that the cosine similarity between two vectors is in the range of [-1,1], and α=0.3.

Images