Dynamic facial expression recognition method for modeling complex time-space relationship

Abstract

The invention relates to a dynamic facial expression recognition method for modeling a complex space-time relationship, and belongs to the field of computer vision. The method comprises the steps of dividing facial expression data into a training set and a verification set, and performing data preprocessing; constructing a multi-view space-time network; and mapping expression spatio-temporal features captured by the multi-view spatio-temporal network to a label system of basic expressions through a full connection layer and a softmax scoring function to complete recognition and classification of facial expressions. According to the method, key spatial features of expressions can be concerned from global and local perspectives, and time sequence features of the expressions are captured from short-term and long-term perspectives, so that a complex space-time relationship generated by individual differences of the expressions is jointly modeled, and the accuracy of a facial expression recognition task is improved.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and relates to a dynamic facial expression recognition method for modeling complex spatiotemporal relationships. Background Art

[0002] Facial expressions are an important way to convey human emotional information and non-verbal communication, and they play an important role in interpersonal communication. Facial Expression Recognition (FER) understands human emotional expressions by capturing changes in facial muscles. Specifically, facial expression features in a given image or video clip are first extracted and mapped to an emotional label system, including seven basic expressions: surprise, sadness, fear, anger, disgust, happiness, and contempt. In recent years, FER, as a hot topic in emotional computing tasks, has been widely used in fields such as human-computer interaction, education, medical care, and driver assistance systems, and has attracted the attention of more and more researchers.

[0003] Existing FER research is mainly divided into static facial expression recognition (SFER) and dynamic facial expression recognition (DFER). Static facial expression recognition usually selects static images at the peak stage of expression as input, and uses deep learning algorithms to learn discriminative feature representations from raw image data for facial expression recognition. Although these methods can achieve high recognition accuracy in some ideal cases using single vertex images, they ignore the dynamic change characteristics of expression. In fact, expression itself is a process that changes dynamically over time, which is completed by one or more movements of facial muscles through multiple stages such as onset, peak and offset. Therefore, DFER has received great attention due to its uniqueness in processing spatial and temporal information. Existing DFER mainly focuses on modeling the spatiotemporal relationship of expression sequences. The simplest method is to directly expand the expression sequence in the time dimension, first connect the output of each frame in the sequence, and then use frame aggregation to generate a fixed-length feature vector for classification. Since the expression intensity in video sequences changes over time, the expression intensity invariant network is proposed to further utilize the dynamic correlation between peak and non-peak expressions to improve performance. In order to encode the temporal dependencies in consecutive frames, a deep spatiotemporal network was proposed. One idea is to add a branch to learn the temporal information of the expression sequence on the basis of extracting spatial features, and learn the spatial and temporal features of facial expressions simultaneously through two parallel branches. Among them, the spatial branch extracts the appearance information of facial expressions through CNN, and the temporal branch captures the motion information on the trajectory of optical flow or facial key points. In order to effectively integrate spatiotemporal features, another idea is to use a serial paradigm, first use CNN to learn the visual representation of each frame of expression, and then use LSTM to process variable length input and output sequences. On the basis of extracting rich spatial features of expressions, further explore the temporal series relationship.

[0004] Although the above methods have made a lot of efforts to explore the spatiotemporal clues of expression sequences, they only consider the spatiotemporal features of a single dimension and ignore the complex spatiotemporal dependencies of expressions, resulting in DFER still facing the following two major challenges. First, the spatial distribution of faces shows obvious individual differences. Since different subjects have unique facial anatomical structures, muscle movement patterns and expression habits, even for the same expression, two subjects will show subtle differences when activated. These differences will affect the appearance and expression effect of the expression, but the local areas of the eyes and mouths of the two are highly similar. Therefore, when extracting expression spatial features, in addition to considering the global information of the expression, more attention should be paid to local key areas to weaken the influence of individual specific attributes. Some existing studies mainly focus on local areas through facial key points. The literature [Zhang K, Huang Y, Du Y, et al. Facial expression recognition based on deep evolutionary alspatial-temporal networks [J]. IEEE Transactions on Image Processing, 2017, 26 (9): 4193-4203.] captures the dynamic changes of facial physical structure by extracting facial key points from consecutive frames. The literature [Qu X, Zou Z, Su X, et al. Attend to where and when: Cascaded attention network for facial expression recognition [J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2021, 6 (3): 580-592.] uses the area within the facial key points as a mask to generate spatial attention and guide the model to focus on the key areas. These methods all require accurate positioning of all facial key points, but some interference factors in complex environments can cause deviations in the positioning of facial key points. Moreover, using only the area within the key points will cause information loss, which is not enough to describe the spatial changes of the face. For example, the upturned corners of the mouth may cause changes in the muscles around the corners of the mouth. We cannot only consider the information within the mouth area and ignore the texture information around it. Therefore, how to reasonably divide the local area of ​​the face is the key to extracting effective spatial features.

[0005] The second challenge is that the temporal evolution of facial expressions also has individual differences. The evolution of facial expressions is completed by the coordinated action of multiple action units. Different combinations of facial action units and movement intensities determine the speed and duration of expression changes, resulting in some expressions being "fast-changing" and some expressions lasting longer. Even if activated by the same subject, different expressions reach their peak at different times. This temporal variability leads to more complex temporal dependencies in dynamic expression sequences, while existing temporal models only focus on short-term or long-term dependencies in expression sequences, making it difficult to capture the complex dynamic features of expression changes. For example, 3DCNN can directly learn dynamic information from short sequences, but cannot obtain long-term dependencies. RNN can gradually transmit and process information at each moment in a long sequence through a gating mechanism. However, it is difficult to pay attention to short-term details when directly using RNN to model time series. With the widespread application of Transformer in computer vision tasks, Vision Transformer (ViT) and its variants use the self-attention mechanism to model the long-term dependencies of image sequences, but its excessive model parameters will increase the computational complexity of the model, and the fixed-size input cannot adapt to the temporal variability of expressions. Therefore, being able to pay attention to both the short-term and long-term temporal information relationships of expression sequences is the key to modeling complex temporal dependencies. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a dynamic facial expression recognition method for modeling complex spatiotemporal relationships, focusing on the key spatial features of expressions from both global and local perspectives, and capturing the temporal features of expressions from short-term and long-term perspectives, so as to jointly model the complex spatiotemporal relationships generated by individual differences in expressions, thereby improving the accuracy of facial expression classification.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A dynamic facial expression recognition method for modeling complex spatiotemporal relationships, the method comprising the following steps:

[0009] S1: Divide the facial expression data into a training set and a validation set, randomly select 16 frames from the original video in chronological order, and then crop the face area;

[0010] S2: Construct a multi-view spatiotemporal network, which includes a multi-region perception module, a multi-time sequence enhancement module, and a classification module;

[0011] S3: The acquired spatiotemporal features of facial expressions are mapped to the label system through a fully connected layer, and then the probability distribution of each emotion category is calculated by the softmax scoring function.

[0012] Furthermore, in S1, for long videos, for training data, 16 expression frames with time differences are randomly selected in the video in chronological order to generate a valid video representation; for test data, 16 frames are uniformly sampled to cover the entire video information, and the sampling formula is as follows:

[0013]

[0014] Among them, frame_index represents the index of the sampled frame in the original video, n represents the nth sampled frame, and video_length represents the total number of frames of the sampled video. After sampling, the face area is cropped out through the face detection algorithm retinaface to eliminate redundant information in the non-face area.

[0015] Further, the S2 comprises the following steps:

[0016] S21: By improving the basic blocks in the Conv4_1 and Conv5_1 layers of ResNet18 into multi-scale convolution blocks, we learn the diverse facial spatial feature representations in the global perception domain of expression. First, we use the shallow expression features F extracted by Conv1 to 3 layers of ResNet18 to n , along the channel axis F n Evenly divide into m feature subsets Then the basic blocks in Conv4_1 and Conv5_1 layers in ResNet18 are improved to multi-scale convolution blocks, so that In addition, each feature subset All undergo a 3×3 convolution operation And each feature subset Convolution with the previous The output of the convolution operation is added and the result is input to the next convolution operation. In the process, convolution iteration operations are performed in sequence; each convolution operation Output It is expressed as follows:

[0017]

[0018] in, From the previous feature subset Get feature information from Compared with the previous feature map It has a larger receptive field; after iterative convolution operations, Contains features of varying numbers and scales;

[0019] Finally, all Output Splicing together, we get a diverse facial spatial feature representation in the global perception domain It is expressed as:

[0020]

[0021] S22: A dynamic segmentation strategy from coarse to fine is proposed to adaptively divide the local key areas of the face, and capture the emotion-related information in each area from the channel and spatial dimensions through attention; first, five key points of eyes, nose and mouth are extracted, and their coordinates are represented as P i =(x i ,y i ), where i∈(eye1,eye2,nose,mouth1,mouth2); then pass through P eye1 and P eye2 , P mouth1 and P mouth2 The midpoint of P nose Generate four local regions (R1, R2, R3, R4); Considering the different distribution of facial features caused by individual attributes, each local region is refined by adaptively reducing the region length. The length of the eye region is r eye and the length of the mouth area r mouth Respectively expressed as:

[0022]

[0023] Among them, x eye1 Indicates P eye1 The horizontal coordinate, x eye Indicates P eye1 and P eye2 The midpoint of mouth1 Indicates P mouth1 The horizontal coordinate, x mouth Indicates P mouth1 and P mouth2 The midpoint of

[0024] Finally, the pixel coordinates of the upper left corner and lower right corner of the corresponding area are calculated by the area length, and the range of the final area (R1, R2, R3, R4) is determined, which is expressed as:

[0025]

[0026] Based on the above local regions, the basic blocks of Conv4_2 and Conv5_2 layers in Resnet18 are improved, and each regional feature map is After two consecutive 3×3 convolution operations, the feature map is obtained Reuse the channel and spatial attention feature map in CBAM The channel and space of the model are weighted to make the model pay more attention to the spatial and channel features related to emotions in the local key area; each local area feature is expressed as:

[0027]

[0028] in, Represents the output of the attention block, Att S Represents spatial attention, Att C represents channel attention, Represents element-wise multiplication;

[0029] To obtain the local features containing four regions Overlay each local feature map along the spatial axis It is expressed as:

[0030]

[0031] Stitching global features and local features Get expression space features It is expressed as:

[0032]

[0033] S23: The temporal difference information of adjacent expression frames is fused by feature reuse to represent the short-term changes of expression; for expression frame n, the corresponding spatial appearance features are first extracted in the short-term window and adjacent frames The difference is then downsampled using the maximum pooling, and then upsampled to match the original features, and finally directly superimposed with the spatial appearance features corresponding to the current frame, the short-term features It is expressed as:

[0034]

[0035] in, Represents a 1D dilated convolution operation with a convolution kernel size of 3, MaxPool represents maximum pooling, and Upsample represents upsampling the feature map describing short-term time information to the same size as the original feature map. The same size to accommodate feature fusion; t = 1 indicates the starting frame, with no short-time differential information;

[0036] S24: Use BiLSTM to model the long-term dependencies of expression sequences in context, and introduce temporal attention to focus on peak expressions with strong discriminability; Based on short-term feature representation BiLSTM is used to obtain the long-term context representation of each frame image feature; BiLSTM contains a forward LSTM and a backward LSTM; the forward LSTM is used to process the features from F1 T arrive The backward LSTM is used to process the features from To F1 T The characteristics of these two processes are expressed as:

[0037]

[0038] in, represents the forward LSTM, represents backward LSTM, and are the outputs of these two processes respectively;

[0039] Will and Connect them together to get the temporal context information of the expression sequence Right now In order to focus on the peak moment of the expression sequence, temporal attention is introduced to compare the strength of frame features at each time step, thereby assigning higher weights to important time steps to obtain the final spatiotemporal feature Y T ; The specific formula is:

[0040]

[0041]

[0042] Among them, f FC (·) represents a fully connected layer, W FC and b FC Represent the weight and bias of the fully connected layer, μ n represents the attention weight at each time step, Represents the weighted features at each time step.

[0043] Furthermore, in S3, a fully connected layer is used to T After dimensionality reduction, a softmax scoring function is added for classification to calculate the probability distribution of each emotion category; the cross entropy loss function is used The difference between the predicted result and the true label is measured and expressed as:

[0044]

[0045] Among them, N represents the number of samples, C represents the number of expression categories, and y i represents the true label, Indicates the prediction result.

[0046] The beneficial effects of the present invention are:

[0047] (1) In the task of dynamic expression recognition, the present invention not only pays attention to the global information of the face, but also adopts an innovative adaptive face local area selection method. This method can effectively extract the features of the local key areas of the face, thereby better adapting to the differences in expression appearance and expression effects between individuals, and enhancing the universality and accuracy of the model for expression recognition of different groups of people.

[0048] (2) Through the feature reuse mechanism, the present invention not only integrates the time difference information between adjacent frames to capture the short-term changes of expressions, but also introduces the ability to capture long-term dependencies through BiLSTM, so that the system can adapt to the complex dynamic changes of expressions. This method of combining short-term and long-term time series analysis significantly improves the ability to recognize the diversity of expression time series and enhances the model's accuracy in interpreting complex emotional changes.

[0049] (3) Existing dynamic expression recognition technologies are usually limited to single-dimensional spatiotemporal modeling, while the present invention innovatively proposes a multi-dimensional spatiotemporal feature aggregation strategy by integrating features from multiple regions and multiple time series. This strategy significantly improves the model's ability to model complex spatiotemporal relationships of expressions in real environments, and enhances expression recognition and classification performance under various interference conditions.

[0050] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0052] Figure 1 A schematic diagram of the dynamic facial expression recognition method for modeling complex spatiotemporal relationships according to the present invention;

[0053] Figure 2 This is a module structure diagram improved by the present invention;

[0054] Figure 3 An illustration of the coarse-to-fine dynamic segmentation strategy proposed in the present invention; (a) determines five facial key points; (b) divides the face into four local areas; (c) refines the boundaries of the four local areas; (d) uses 0 filler to fix the size. DETAILED DESCRIPTION

[0055] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0056] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0057] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0058] See also Figure 1 to Figure 3 , is a dynamic facial expression recognition method that models complex spatiotemporal relationships. This method is mainly used for dynamic facial expression recognition in real scenes. Technical problem solved: Facial expressions show individual differences in spatiotemporal dimensions, leading to the problem of emotion misclassification.

[0059] S1: Divide the facial expression data into a training set and a validation set, randomly select 16 frames from the original video in chronological order, and then crop the face area;

[0060] S2: Construct a multi-view spatiotemporal network, which includes a multi-region perception module, a multi-time sequence enhancement module, and a classification module;

[0061] S3: The acquired spatiotemporal features of facial expressions are mapped to the label system through a fully connected layer, and then the probability distribution of each emotion category is calculated by the softmax scoring function.

[0062] Furthermore, in S1, for long videos, since adjacent frames are very similar in a short time window, it is inefficient to directly stack multiple frames for subsequent processing. Therefore, for the training data, 16 expression frames with time differences are randomly selected in the video in chronological order to generate an effective video representation. For the test data, in order to cover the entire video information, 16 frames are uniformly sampled, and the sampling formula is as follows:

[0063]

[0064] Among them, frame_index represents the index of the sampled frame in the original video, n represents the nth sampled frame, and video_length represents the total number of frames of the sampled video. After sampling, the face detection algorithm retinaface is used to crop the face area to eliminate redundant information in the non-face area.

[0065] Further, in S2, constructing a multi-view spatiotemporal network comprises the following steps:

[0066] S21: By improving the basic blocks in Conv4_1 and Conv5_1 layers in ResNet18 (such as Figure 2 (a) shows a multi-scale convolutional block that learns diverse facial spatial feature representations within the global perception domain of expression. Figure 2 As shown in (b). First, the shallow expression features F extracted by Conv1 to 3 layers of ResNet18 are used to extract the facial expressions. n , along the channel axis F n Evenly divide into m feature subsets Then the basic blocks in Conv4_1 and Conv5_1 layers in ResNet18 are improved to multi-scale convolution blocks, so that In addition, each feature subset All undergo a 3×3 convolution operation And each feature subset Convolution with the previous The output of the convolution operation is added and the result is input to the next convolution operation. In the process, convolution iterations are performed in sequence. Each convolution operation Output It is expressed as follows:

[0067]

[0068] in, All of them may be from the previous feature subset Get feature information from Compared with the previous feature map have a larger receptive field. Therefore, after the convolution iteration operation, Contains features of varying numbers and scales.

[0069] Finally, all Output Splicing together, we get a diverse facial spatial feature representation in the global perception domain It can be expressed as:

[0070]

[0071] S22: A dynamic segmentation strategy from coarse to fine is proposed to adaptively divide the local key areas of the face, and capture the emotion-related information in each area from the channel and spatial dimensions through attention, such as Figure 3 As shown in the figure, (a) determines five facial key points; (b) divides four local face regions; (c) refines the boundaries of the four local regions; (d) uses 0 filler to fix the size. First, extract the five key points of the eyes, nose and mouth, and their coordinates are represented by P i =(x i ,y i ), where i∈(eye1,eye2,nose,mouth1,mouth2). Then pass through P eye1 and P eye2 , P mouth1 and P mouth2 The midpoint of P nose Four local regions (R1, R2, R3, R4) are generated. Taking into account the different distribution of facial features caused by individual attributes, each local region is further refined by adaptively reducing the region length. The length of the eye region is r eye and the length of the mouth area r mouth They can be expressed as:

[0072]

[0073] Among them, x eye1 Indicates P eye1 The horizontal coordinate, x eye Indicates P eye1 and P eye2 The midpoint of mouth1 Indicates P mouth1 The horizontal coordinate, x mouth Indicates P mouth1 and P mouth2 The midpoint of .

[0074] Finally, the pixel coordinates of the upper left corner and lower right corner of the corresponding area are calculated by the area length, and the range of the final area (R1, R2, R3, R4) is determined, which is expressed as:

[0075]

[0076] Based on the above local areas, such as Figure 2 As shown in (c), the basic blocks of Conv4_2 and Conv5_2 layers in Resnet18 are improved, and each regional feature map is After two consecutive 3×3 convolution operations, the feature map is obtained Reuse the channel and spatial attention feature map in CBAM The channel and space of the attention weighting are weighted to make the model pay more attention to the spatial and channel features related to emotions in the local key area. Therefore, each local area feature can be expressed as:

[0077]

[0078] in, Represents the output of the attention block, Att S Represents spatial attention, Att C represents channel attention, Represents element-wise multiplication.

[0079] In order to obtain the local features containing four regions Overlay each local feature map along the spatial axis It is expressed as:

[0080]

[0081] Stitching global features and local features Expression space features can be obtained It is expressed as:

[0082]

[0083] S23: Fuse the temporal difference information of adjacent expression frames by feature reuse to represent the short-term changes of expression. For expression frame n, first extract its corresponding spatial appearance features in the short-term window and adjacent frames The difference is then downsampled using the maximum pooling, and then upsampled to match the original features, and finally directly superimposed with the spatial appearance features corresponding to the current frame, the short-term features It can be expressed as:

[0084]

[0085] in, Represents a 1D dilated convolution operation with a convolution kernel size of 3, MaxPool represents maximum pooling, and Upsample represents upsampling the feature map describing short-term time information to the same size as the original feature map. The same size is used to accommodate feature fusion. Since t = 1 represents the starting frame, there is no short-term differential information.

[0086] S24: Use BiLSTM to model the long-term dependencies of expression sequences in context, and introduce temporal attention to focus on peak expressions with strong discriminability. BiLSTM is used to obtain the long-term context representation of each frame image feature. Among them, BiLSTM contains a forward LSTM (processing from F1 T arrive features) and a backward LSTM (processing from To F1 T The two processes can be expressed as:

[0087]

[0088] in, represents the forward LSTM, represents backward LSTM, and are the outputs of these two processes respectively.

[0089] Will and Connecting them together can get the temporal context information of the expression sequence Right now In order to focus on the peak moment of the expression sequence, temporal attention is introduced to compare the strength of frame features at each time step, thereby assigning higher weights to important time steps to obtain the final spatiotemporal feature Y T The specific formula is:

[0090]

[0091] Among them, f FC (·) represents a fully connected layer, W FC and b FC Represent the weight and bias of the fully connected layer, μ n represents the attention weight at each time step, Represents the weighted features at each time step.

[0092] Furthermore, in S3, a fully connected layer is used to T After dimensionality reduction, a softmax scoring function is added for classification to calculate the probability distribution of each emotion category. In the embodiment, the cross entropy loss function is used. The difference between the predicted result and the true label is measured and expressed as:

[0093]

[0094] Among them, N represents the number of samples, C represents the number of expression categories, and y i represents the true label, Indicates the prediction result.

[0095] Example 1

[0096] This example compares the proposed method with existing methods based on CK+, Oulu-CASIA and MMI datasets.

[0097] CK+ is an extension of the Cohn-Kanade dataset and includes 593 facial expression sequences from 123 subjects. These sequences have a duration between 10 and 60 frames and record the transition from neutral to peak emotion. Among these sequences, 327 sequences from 118 subjects are annotated with seven facial expressions (anger, contempt, disgust, fear, happiness, sadness, and surprise). The Oulu-CASIA dataset contains 2880 expression sequences from 80 subjects aged between 23 and 58 years old. These sequences are labeled with six basic emotion categories: anger, disgust, fear, happiness, sadness, and surprise. Similar to the CK+ dataset, these expression sequences show the process of changing from neutral to peak emotion. The MMI dataset contains expression sequences of 326 different subjects from 30 different regions (Europe, Asia, and South America) and aged between 19 and 62 years old. Among these sequences, 213 sequences are labeled with six basic emotions (anger, disgust, fear, happiness, sadness, and surprise), and 205 sequences show frontal views of facial expressions. Different from the emotional change trends presented in the first two datasets, the MMI dataset starts from neutral, reaches the emotional peak, and then returns to a neutral state.

[0098] This embodiment compares the recognition performance of the proposed method with other facial expression recognition methods on the CK+ dataset. Table 1 is the comparison result of the recognition performance (%) of the proposed method. Table 1 shows that the proposed method achieves superior performance with pre-training, with an accuracy of 99.14%. Compared with other methods, the method of the present invention shows significant superiority due to the consideration of individual specificity. In the dynamic method, ESTLNet constructs global and local dual channels through the attention mechanism, achieving an optimal recognition accuracy of 99.04%. However, relying solely on attention to select local features may not be accurate. The difference is that the proposed method can adaptively focus on the significant features of the local area based on individual features, improving the accuracy by 0.10%. Graph-Tran uses facial key points and their corresponding local areas to construct a spatiotemporal face graph, aggregates local and non-local information in the graph structure, and achieves an accuracy of 98.87%. In contrast, the recognition accuracy of the method proposed in the present invention is improved by 0.27%, which is attributed to the fact that the proposed method explicitly constructs multi-scale global context information and considers more effective information such as texture around facial key points. However, compared with the best static method PPDN, the accuracy of the proposed method is reduced by 0.16%. This is because the CK+ dataset is a high-quality dataset collected in a controlled environment in the implementation room, and PPDN can reduce the interference of redundant information by manually selecting expression images at the peak stage as input.

[0099] Table 1

[0100]

[0101] This embodiment compares the recognition performance of the proposed method with other facial expression recognition methods on the Oulu-CASIA dataset. Table 2 shows the comparison results of the recognition performance (%) of the proposed method. Table 2 shows that the proposed method achieves an accuracy of 91.67% on the Oulu-CASIA dataset, which is better than the most advanced static and dynamic methods. Unlike the performance on the CK+ dataset, the proposed method significantly improves the recognition accuracy by 7.08% compared with the optimal static method PPDN. The reason may be that the Oulu-CASIA dataset contains more interference factors, such as changing head postures and faces blocked by glasses. In addition, the proposed method applies a dynamically adjusted local perception mechanism to maximize the inclusion of key information and reduce interference. Compared with the optimal dynamic method ESTLNet and the suboptimal dynamic method Graph-Tran, the performance of the proposed method is significantly improved by 2.29% and 2.50%, which is a great improvement compared with the CK+ dataset. The results show that the proposed method has an advantage in processing more complex expression sequences because it adaptively adjusts the region of interest or time step in the spatial and temporal dimensions according to individual characteristics, rather than relying solely on the self-learning of the model.

[0102] Table 2

[0103]

[0104]

[0105] This embodiment compares the recognition performance of the proposed method with other facial expression recognition methods on the MMI dataset. Table 3 is the comparison result of the recognition performance (%) of the proposed method. Table 3 shows that the proposed method achieves the best performance (85.33%) on the MMI dataset. Compared with IF-GAN (a method that uses synthetic average faces to eliminate individual specific attributes), a better performance is achieved. This shows that the proposed method has better generalization ability for individual specificity due to its focus on highly similar eye and mouth areas. Compared with the dynamic methods PHRNN-MSCNN and CAN that also use facial key points, the accuracy of the proposed method is improved by 4.15% and 1.78%, respectively. PHRNN-MSCNN requires a large number of facial key points to capture the dynamic changes of the face, and CAN only weights the information within the area composed of facial key points. However, the proposed method only applies five facial key points to adaptively divide the key area and consider the surrounding texture information, thereby effectively reducing interference and capturing as many effective features as possible.

[0106] Table 3

[0107]

[0108] Example 2

[0109] This embodiment is an ablation embodiment of the method proposed by the present invention.

[0110] First, the influence of different components in the multi-region perception module is explored. The multi-region perception module is a dual-branch structure that integrates global and local spatial feature extraction branches. The present invention has implemented embodiments on each branch to evaluate the contribution of different components. One Branch represents a single-branch structure with ResNet18 as the backbone. TB represents a dual-branch structure, in which the Conv4_1 and Conv5_1 layers constitute one branch, and the Conv4_2 and Conv_5_2 layers constitute another branch. In order to prove the effectiveness of global features, the present invention replaces one branch of TB with a global spatial feature extraction branch composed of multi-scale convolution blocks, represented by Baseline+TB+GB. In order to verify the effectiveness of local features, the present invention replaces another branch of TB with a local spatial feature extraction branch, which is composed of a coarse-to-fine dynamic partitioning strategy and a regional perception attention block, represented by Baseline+TB+GB+LB.

[0111] Table 4 shows the performance of different branches of the multi-region perception module. Compared with the Baseline, Baseline+TB has different degrees of improvement on the three datasets, which shows that the dual-branch structure has a stronger feature representation ability. Compared with Baseline+TB, the recognition results of Baseline+TB+GB on the three datasets are improved by 0.92%, 0.83% and 0.47%, respectively. The reason is that the global feature extraction branch further establishes the connection between different scale features in the global perception domain. Compared with Baseline+TB+GB, the recognition results of Baseline+TB+GB+LB on the three datasets are improved by 0.91%, 1.46% and 0.93%, respectively, which shows that the local feature extraction branch can focus on key facial areas to extract effective emotional information. On the CK+, Oulu-CASIA and MMI datasets, the multi-region perception module is improved by 1.85%, 2.50% and 1.88% compared with the Baseline, respectively. The results show that the dual-branch structure of the multi-region perception module can learn effective spatial feature representation from both global and local perspectives.

[0112] Table 4

[0113] Model CK+ Oulu-CASIA MMI Baseline(OneBranch) 96.02 86.46 81.52 Baseline+TB 96.04 86.67 82.00 Baseline+TB+GB 96.96 87.50 82.47 Baseline+TB+GB+LB 97.87 88.96 83.40

[0114] Then, the influence of different components in the multi-temporal enhancement module is explored. A large number of embodiments are carried out on the CK+, Oulu-CASIA and MMI datasets to evaluate the effectiveness of the multi-temporal enhancement module, and the results are shown in Tables 3 to 5. Specifically, BiLSTM indicates whether bidirectional LSTM is used, STB indicates short-term block, and TA indicates temporal attention.

[0115] As shown in Table 5, the application of BiLSTM enables the multi-temporal enhancement module to achieve significant performance improvements on the CK+, Oulu-CASIA, and MMI datasets, which is attributed to its ability to capture bidirectional semantics in the temporal information of dynamic expression sequences. In addition, on the CK+, Oulu-CASIA, and MMI datasets, the recognition results are improved by 0.61%, 0.83%, and 0.55% accuracy by superimposing STB on BiLSTM, respectively. The results show that the short-term block is able to capture effective short-term motion features from variable temporal information for temporal modeling of expressions. By adding TA to assign attention weights at different time steps, the recognition accuracy on the three datasets is improved by 0.31%, 0.84%, and 0.48%, respectively, which shows that TA can automatically capture the information of the peak frame. Compared with the baseline, the performance of BiLSTM, STB, and TA in the multi-temporal enhancement module on the CK+, Oulu-CASIA, and MMI datasets is significantly improved by 1.21%, 2.30%, and 1.05%, respectively.

[0116] Table 5

[0117]

[0118]

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A dynamic facial expression recognition method for modeling complex spatiotemporal relationships, characterized in that: The method comprises the following steps: S1: Divide the facial expression data into a training set and a validation set, randomly select 16 frames from the original video in chronological order, and then crop the face area; S2: Construct a multi-view spatiotemporal network, which includes a multi-region perception module, a multi-time sequence enhancement module, and a classification module; S3: The acquired spatiotemporal features of facial expressions are mapped to the label system through a fully connected layer, and then the probability distribution of each emotion category is calculated by the softmax scoring function.

2. A method for dynamic facial expression recognition based on modeling complex spatiotemporal relationships according to claim 1, characterized in that: In S1, for a long video, for training data, 16 expression frames with time differences are randomly selected in the video in chronological order to generate a valid video representation; for test data, 16 frames are uniformly sampled to cover the entire video information, and the sampling formula is as follows: Among them, frame_index represents the index of the sampled frame in the original video, n represents the nth sampled frame, and video_length represents the total number of frames of the sampled video. After sampling, the face area is cropped out through the face detection algorithm retinaface to eliminate redundant information in the non-face area.

3. A method for dynamic facial expression recognition based on modeling complex spatiotemporal relationships according to claim 2, characterized in that: The S2 comprises the following steps: S21: By improving the basic blocks in the Conv4_1 and Conv5_1 layers of ResNet18 into multi-scale convolution blocks, we learn the diverse facial spatial feature representations in the global perception domain of expression. First, we use the shallow expression features F extracted by Conv1 to 3 layers of ResNet18 to n , along the channel axis F n Evenly divide into m feature subsets Then the basic blocks in Conv4_1 and Conv5_1 layers in ResNet18 are improved to multi-scale convolution blocks, so that In addition, each feature subset All go through a 3×3 convolution operation And each feature subset Convolution with the previous The output of the convolution operation is added and the result is input to the next convolution operation. In the process, convolution iteration operations are performed in sequence; each convolution operation Output It is expressed as follows: in, From the previous feature subset Get feature information from Compared with the previous feature map It has a larger receptive field; after iterative convolution operations, Contains features of varying numbers and scales; Finally, all Output Splicing together, we get a diverse facial spatial feature representation in the global perception domain It is expressed as: S22: A dynamic segmentation strategy from coarse to fine is proposed to adaptively divide the local key areas of the face, and capture the emotion-related information in each area from the channel and spatial dimensions through attention; first, five key points of eyes, nose and mouth are extracted, and their coordinates are represented as P i =(x i ,y i ), where i∈(eye1,eye2,nose,mouth1,mouth2); then pass through P eye1 and P eye2 , P mouth1 and P mouth2 The midpoint of P nose Generate four local regions (R1, R2, R3, R4); Considering the different distribution of facial features caused by individual attributes, each local region is refined by adaptively reducing the region length. The length of the eye region is r eye and the length of the mouth area r mouth Respectively expressed as: Among them, x eye1 Indicates P eye1 The horizontal coordinate, x eye Indicates P eye1 and P eye2 The midpoint of mouth1 Indicates P mouth1 The horizontal coordinate, x mouth Indicates P mouth1 and P mouth2 The midpoint of Finally, the pixel coordinates of the upper left corner and lower right corner of the corresponding area are calculated by the area length, and the range of the final area (R1, R2, R3, R4) is determined, which is expressed as: Based on the above local regions, the basic blocks of Conv4_2 and Conv5_2 layers in Resnet18 are improved, and each regional feature map is After two consecutive 3×3 convolution operations, the feature map is obtained Reuse the channel and spatial attention feature map in CBAM The channel and space of the model are weighted to make the model pay more attention to the spatial and channel features related to emotions in the local key area; each local area feature is expressed as: in, Represents the output of the attention block, Att S Represents spatial attention, Att C represents channel attention, Represents element-wise multiplication; To obtain the local features containing four regions Overlay each local feature map along the spatial axis It is expressed as: Stitching global features and local features Get expression space features It is expressed as: S23: The temporal difference information of adjacent expression frames is fused by feature reuse to represent the short-term changes of expression; for expression frame n, the corresponding spatial appearance features are first extracted in the short-term window and adjacent frames The difference is then downsampled using the maximum pooling, and then upsampled to match the original features, and finally directly superimposed with the spatial appearance features corresponding to the current frame, the short-term features It is expressed as: in, Represents a 1D dilated convolution operation with a convolution kernel size of 3, MaxPool represents maximum pooling, and Upsample represents upsampling the feature map describing short-term time information to the same size as the original feature map. The same size to accommodate feature fusion; t = 1 indicates the starting frame, with no short-time differential information; S24: Use BiLSTM to model the long-term dependencies of expression sequences in context, and introduce temporal attention to focus on peak expressions with strong discriminability; Based on short-term feature representation BiLSTM is used to obtain the long-term context representation of each frame image feature; BiLSTM contains a forward LSTM and a backward LSTM; the forward LSTM is used to process the F1 T arrive The backward LSTM is used to process the features from To F1 T The characteristics of these two processes are expressed as: in, represents the forward LSTM, represents backward LSTM, and are the outputs of these two processes respectively; Will and Connect them together to get the temporal context information of the expression sequence Right now In order to focus on the peak moment of the expression sequence, temporal attention is introduced to compare the strength of frame features at each time step, thereby assigning higher weights to important time steps to obtain the final spatiotemporal feature Y T ; The specific formula is: Among them, f FC (·) represents a fully connected layer, W FC and b FC Represent the weight and bias of the fully connected layer, μ n represents the attention weight at each time step, Represents the weighted features at each time step.

4. A method for dynamic facial expression recognition based on modeling complex spatiotemporal relationships according to claim 3, characterized in that: In S3, a fully connected layer is used to T After dimensionality reduction, a softmax scoring function is added for classification to calculate the probability distribution of each emotion category; the cross entropy loss function is used The difference between the predicted result and the true label is measured and expressed as: Among them, N represents the number of samples, C represents the number of expression categories, and y i represents the true label, Represents the prediction result.

Images