A video expression recognition method based on space-time feature decoupling and application

Abstract

The application discloses a video expression recognition method based on space-time feature decoupling, and steps of the method comprise the following steps: 1, obtaining a data set in a laboratory environment and a data set in a natural environment, and performing pretreatment; 2, obtaining expression detail features of important regions of facial expressions through a global feature extraction region layer network; 3, putting global facial expression features extracted by the global feature extraction region layer network into a RoI feature extraction network to perform RoI feature extraction; 4, making the model perform decoupling operation in two dimensions of time and space through a space-time feature decoupling module; 5, obtaining label prediction probability through a classification module; 6, correcting wrong sample labels in the data set through a label correction module; and 7, obtaining a final emotion recognition result through the label prediction probability. The application can significantly improve a video emotion recognition rate.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of video computing, and particularly relates to a video expression recognition method based on space-time feature decoupling and application. BACKGROUND

[0002] In recent years, the transformer has achieved great success in the field of natural language processing. The self-attention mechanism in the transformer can obtain the global dependency relationship between each input element, which is very suitable for the facial expression recognition task, because of the complexity of the expression form. This requires the model to obtain the internal relationship of each important facial region. For example, some difficult-to-distinguish or ambiguous facial images, such as 'laughing and crying','mixing joy and sorrow', and some complex facial expressions. In this regard, the self-attention mechanism can make the model flexibly handle the correlation between the important regions of the facial expression. However, the input of the transformer self-attention mechanism is sequence data. In the video sequence, the features of the RoI extracted from each video frame contain both spatial and temporal dimensions. If the space-time feature information is directly unfolded into a one-dimensional vector, the temporal and spatial features will be confused, causing the semantic confusion of the feature information and affecting the recognition performance of the model. In addition, directly unfolding will increase the length of the feature sequence, and considering the dot product operation in the self-attention mechanism, this operation greatly increases the computational complexity.

[0003] Research has found that the occurrence of human facial expressions is usually completed by the muscle movements of the eyebrow, mouth and other regions. How to make the model focus on the movement change features of these regions can help the model exclude the interference of the regions unrelated to the expression in the face and improve the efficiency of feature extraction of the model.

[0004] Some data labels in some public data sets have certain deviations, which are mainly caused by the subjectivity of the label annotators, which may cause the model to learn incorrect features during training, greatly reducing the recognition accuracy and robustness of the model. SUMMARY

[0005] The application is proposed to solve the above-mentioned problems of the prior art, and provides a video expression recognition method based on space-time feature decoupling and application, so as to fully utilize the expression semantic information and the detailed feature information of the facial expression in the video sequence, thereby significantly improving the video emotion recognition rate.

[0006] In order to achieve the above-mentioned application purposes, the application adopts the following technical solutions.

[0007] The video expression recognition method based on space-time feature decoupling has the following characteristics:

[0008] Step one, obtaining a video expression dataset with label annotation in an indoor environment, and performing length standardization processing on each video in the video expression dataset, and then uniformly processing the size of T-frame face images in each standardized video, thereby obtaining a pre-processed video expression sequence corresponding to each video;

[0009] using the Dlib tool to mark the face feature points of the t-th frame face image x t in the pre-processed video expression sequence Q corresponding to any one video, thereby according to the face feature points, using a face motion unit segmentation rule to divide the face region in the t-th frame face image x t , to obtain a face region set Z t of the t-th frame face image x t , thereby obtaining a face region set Z of the video expression sequence Q;

[0010] Step two, constructing an expression recognition network, including: a global feature extraction region layer network, an RoI feature extraction network, a space-time feature decoupling network, a classification module and a label correction module;

[0011] Step 2.1, the global feature extraction region layer network is composed of the first three modules of the SE-ResNet-50 network and a region layer;

[0012] The video expression sequence Q is input into the global feature extraction region layer network, and after feature extraction by the first three modules, a feature map sequence f Q ={f1,…,f t ,…,f T} of the video expression sequence Q is obtained, wherein f t represents the feature map of the t-th frame face image x t , and T represents the frame number of the video expression sequence Q;

[0013] The region layer first uniformly divides the feature map f t of the t-th frame face image x t into i×i image blocks, and then performs normalization and ReLU activation function operation on each image block, thereby obtaining i×i processed image blocks;

[0014] The region layer uses a C-channel convolution layer with a convolution kernel of 1×1 to perform local convolution operation on each processed image block, to obtain a feature vector of each image block with C channels, thereby obtaining a feature channel vector set b t ;

[0015] The region layer uses two fully connected layers and a sigmoid function in turn to process the feature channel vector set bt The feature channel inter-correlation calculation is performed to obtain the weight corresponding to each feature channel, and the weight corresponding to each feature channel is multiplied and weighted with the processed i x i image blocks respectively, so as to obtain an i x i image block set a with V feature channels and weights t

[0016] The image block set a t is matched with the feature map f t The position of the i x i image block is matched, and the global expression information weight feature map P t of the feature map f t is obtained, so as to obtain a global expression information weight feature map sequence P Q ={P1, …, P t , …, P T};

[0017] Step 2.2, the RoI feature extraction network is composed of the fourth SE-Res module in the SE-ResNet-50 network; and the face region in the face region set Z is taken as the recognition boundary of the face image of the fourth SE-Res module;

[0018] The fourth SE-Res module uses a pooling layer to obtain n RoI feature maps from the global expression information weight feature map P t and perform size unification, so as to obtain the RoI feature map set R t ={R t , …, R t1 , …, P tk} of the t-th frame face image x tn , wherein R tk represents the k-th size-unified RoI feature map, and n represents the number of RoI feature maps;

[0019] The fourth SE-Res module performs fine-grained feature extraction on the RoI feature map set R t , so as to obtain the high-level semantic feature G t ={G t1 , …, G tk , …, G tn} of the n RoI feature maps, wherein G tk represents the high-level semantic feature of the k-th size-unified feature map R tk , so as to obtain a high-level semantic feature sequence G Q ={G1, …, G t , …, G T};

[0020] ​Step 2.3, the space-time feature decoupling network, in turn comprises: L layers of spatial attention modules, L layers of temporal attention modules, a global average pooling layer and a fully connected layer:

[0021] Step 2.3.1, each layer of spatial attention module in turn consists of S attention heads, a linear layer, a normalization layer BN, and an activation function layer ReLU;

[0022] a sequence of high-level semantic features G Q is input into the first layer of spatial attention module, and S attention heads are used to calculate the spatial attention feature map of G Q , to obtain S spatial attention feature maps of the t-th frame of high-level semantic features G ti . Thus, the spatial attention feature map of the sequence of high-level semantic features G Q is obtained. wherein, the i-th attention head in the first layer of spatial attention module first encodes the spatial position of G t according to the position of each RoI in G t , and then adds the spatial position encoding to the t-th frame of high-level semantic features G t , and after linear function operation, the i-th attention head of the t-th frame of high-level semantic features G ti with spatial position encoding is obtained, so as to calculate the spatial attention feature map of the t-th frame of high-level semantic features G ti with spatial position encoding by formula (1). And the spatial attention feature map A 1i of the i-th attention head of the first layer of spatial attention module is obtained by formula (2):

[0023]

[0024]

[0025] In formula (1), σ and φ are two embedding functions, (G ti )' represents the transpose of the high-level semantic feature G ti ; softmax represents a normalized exponential function, and tanh represents an activation function.

[0026] After multiplying the spatial attention feature map A 1i output by the i-th attention head of the first layer of spatial attention module with the sequence of high-level semantic features G Q , a matrix is obtained. Thus, the spatial attention feature maps of the S attention heads are obtained respectively. QThe multiplied matrix is spliced, and then sequentially processed by a linear layer, a normalization layer BN, and an activation function layer ReLU, to obtain a spatial output matrix

[0027] Step 2.3.2, the spatial output matrix is transposed into a time matrix Y Q , and each time element is a matrix of n C channels, wherein each C channel represents a time set of a high-level semantic feature, so that the n C channels are regarded as n time sets, denoted as Y Q = {Y1,..., Y k ,..., Y n}; wherein Y k represents the time set of the kth high-level semantic feature.

[0028] Step 2.3.3, each layer of the time attention module has S attention heads, a linear layer, a normalization layer BN, and an activation function layer ReLU in sequence.

[0029] The time matrix Y Q is input into the first layer of the time attention module, and the S attention heads perform time attention feature map calculation on Y Q to obtain the time attention feature map of the S attention heads , thereby obtaining the time attention feature map of the matrix Y , wherein the i th attention head of the first layer of the time attention module first performs time position coding on Y k according to the frame number order in the video expression sequence Q, and adds the time position coding to Y k of the k th C channel, and then obtains the time set Y ki of the k th high-level semantic feature with time position coding of the i th attention head after linear function operation, thereby calculating the time attention feature map of the time set Y ki of the k th high-level semantic feature with time position coding of the video expression sequence Q by using formula (3) , and obtaining the time attention feature map B 1i of the i th attention head of the first layer of the time attention module by using formula (4):

[0030]

[0031]

[0032] In formula (3), (Y ki )' represents the transpose of Y ki .

[0033] The time attention feature map B output by the i-th attention head of the first-layer time attention module. 1i With matrix Y Q The matrix is ​​obtained after multiplication. Thus, the temporal attention feature maps of S attention heads are obtained and compared with matrix Y. Q The multiplied matrices are concatenated and then processed sequentially through a linear layer, a normalization layer (BN), and an activation function layer (ReLU) to obtain the time output matrix.

[0034] Step 2.3.4: Output the time matrix The matrix is ​​transformed into a spatial matrix with n spatial elements, each having T C channels. Following steps 2.3.1-2.3.3, the n spatial elements are sequentially processed by the next layer of spatial attention module and temporal attention module, resulting in a temporal output matrix after being processed by L layers of spatial attention modules and L layers of temporal attention modules.

[0035] Step 2.3.5: Output the time matrix After passing through a global average pooling layer and a fully connected layer in sequence, a one-dimensional vector α is obtained;

[0036] Step 2.4: Input the one-dimensional vector α into the classification module consisting of two fully connected layers and a Softmax function for processing to obtain the label prediction probability of the video expression sequence Q;

[0037] Step 2.5: The label correction module includes: a self-attention weight module, a weight ranking module, and a label re-labeling module;

[0038] The video facial expression sequence Q is processed by the self-attention weighting module to obtain a weight score a. Q ;

[0039] The weighted ranking module sorts the weight scores a according to the set weight standard value λ. Q Classified into high-weight restructuring or low-weight restructuring;

[0040] If the weight score of the video expression sequence Q is a Q In low-weight recombination, and the maximum prediction probability P in the label prediction probability vector. max The corresponding tag g max If the difference between Q and its true label is greater than the set threshold ξ, then Q's label will be relabeled as the maximum predicted probability P. max The tag g corresponding to the rate max Otherwise, preserve the original labels of the Q-sequence of the video expressions;

[0041] Step three, the maximum value corresponding to the emotion category in the label prediction probability vector of the video expression sequence Q is taken as the final emotion classification result.

[0042] The electronic device of the present application comprises a memory and a processor, characterized in that the memory is used to store a program supporting the processor to execute the video expression recognition method, and the processor is configured to execute the program stored in the memory.

[0043] The computer readable storage medium of the present application stores a computer program, characterized in that the computer program is executed by the processor to execute the steps of the video expression recognition method.

[0044] Compared with the prior art, the present application has the following advantages:

[0045] 1. The present application adds a region layer network to the global feature network of facial expression to obtain the expression detail features of the important region of facial expression, thereby overcoming the problem of strong similarity between expression forms in facial expression recognition, and solving the problem of inability to capture the local and subtle appearance changes of the important region of facial expression due to the structural nature of facial features, and finally obtaining the expression detail features of the important region of facial expression efficiently and accurately.

[0046] 2. The present application decouples the time and space dimensions to greatly reduce the model calculation complexity on the basis of improving the model recognition performance, and solves the problem of ignoring the difference between the two dimensions in the previous method due to the same processing of the features of the two dimensions. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The video expression recognition network of the present application decouples space-time features;

[0048] Figure 2 The feature point numbering diagram of the present application;

[0049] Figure 3 The RoI region diagram corresponding to the facial motion unit group of the present application;

[0050] Figure 4 The region layer network structure diagram of the present application;

[0051] Figure 5 The space-time feature decoupling network diagram of the present application;

[0052] Figure 6 The spatial attention module structure diagram of the present application;

[0053] Figure 7A schematic diagram of the spatio-temporal feature decoupling scheme of the present application;

[0054] Figure 8 A label correction network diagram of the present application. DETAILED DESCRIPTION

[0055] In this embodiment, a video expression recognition method based on spatio-temporal feature decoupling is introduced to fully utilize the expression semantic information contained in the video sequence by introducing a self-attention mechanism into the model. Meanwhile, considering the specificity of spatio-temporal features, a spatio-temporal feature decoupling network is designed to ensure the independence of time and spatial dimension features. In addition, to enhance the representation ability of the model for facial expressions, a region layer network is added to the model to obtain detailed feature information of facial expressions. The overall process of the method is shown in Figure 1 , and the steps include: obtaining a video expression dataset with label annotation in an indoor environment, and pre-processing, facial motion unit division and grouping; then establishing a global feature network, adding a region layer network in the global feature network of facial expressions to obtain expression detail features of important regions of facial expressions; then putting the global features extracted by the global feature extraction network into the RoI feature extraction network for RoI feature extraction; then decoupling the features through the spatio-temporal feature decoupling module so that the model can specifically process the features in the time and space dimensions; then obtaining the label prediction probability through the classification module; correcting the wrong sample labels in the dataset through the label correction module; finally, obtaining the final emotion recognition result through the label prediction probability. The present application can efficiently and accurately obtain the expression detail features of important regions of facial expressions, greatly reduce the model calculation complexity on the basis of improving the model recognition performance, and significantly improve the accuracy of video expression recognition. Specifically, the steps include:

[0056] Step one, obtaining MMI and Oulu-CASIA video expression datasets with label annotation in an indoor environment, and performing length standardization processing on each video in the video expression dataset, and then uniformly processing the size of T=16 frames of facial images in each standardized video to 224x224, thereby obtaining the pre-processed video expression sequence corresponding to each video;

[0057] Using the Dlib tool to mark the facial feature points of the t-th frame of facial image x t in the pre-processed video expression sequence Q corresponding to any one video, 43 feature points are obtained as shown in Figure 2 , thereby dividing the facial region in the t-th frame of facial image x t according to the facial motion unit segmentation rule (see Table 1), and the face is divided into 9 regions, and the facial region set Z of the t-th frame of facial image x t is obtainedt , thereby obtaining a face region set Z of the video expression sequence Q, as shown in Figure 3 ;

[0058] Table 1 Face movement unit segmentation rule

[0059]

[0060]

[0061] Step two, constructing an expression recognition network, including: a global feature extraction region layer network, an RoI feature extraction network, a space-time feature decoupling network, a classification module and a label correction module;

[0062] Step 2.1, the global feature extraction region layer network is composed of the first three modules of the SE-ResNet-50 network and a region layer, and the region layer network is used to obtain expression detail features of important regions of facial expressions. The network structure of the region layer is as shown in Figure 4 , which mainly includes three steps of image block cropping, local convolution and identity addition.

[0063] The video expression sequence Q is input into the global feature extraction region layer network, and after feature extraction by the first three modules, a feature map sequence f Q = {f1, …, f t , …, f T} of the video expression sequence Q is obtained, wherein f t represents a feature map of the t-th frame face image x t , and T represents the frame number of the video expression sequence Q.

[0064] The region layer first divides the feature map f t of the t-th frame face image x t into 8x8 image blocks, and then performs normalization and ReLU activation function operation on each image block, thereby obtaining 8x8 processed image blocks.

[0065] The region layer uses a convolution layer composed of C=64 1x1 convolution kernels to perform local convolution operation on each processed image block, thereby obtaining a feature vector of each image block with C=64 channels, and thereby obtaining a set b t composed of feature channel vectors of 8x8 image blocks.

[0066] The region layer uses two full connection layers and a sigmoid function in sequence to process the feature channel vector set b tThe feature channel inter-correlation calculation is performed to obtain the weight corresponding to each feature channel, and the weight corresponding to each feature channel is multiplied by the processed 8x8 image block respectively to obtain an 8x8 image block set a containing 64 feature channels with weights t ;

[0067] The image block set a t is matched with the feature map f t The position of the upper 8x8 image block is matched to obtain the global expression information weight feature map P t of the feature map f t , thereby obtaining the global expression information weight feature map sequence P Q ={P1, …, P t , …, P T}, the global expression information weight feature map assigns a higher weight to the image block containing a large amount of facial expression feature information, and a lower weight to the image block containing a small amount of facial expression feature information, thereby realizing feature enhancement on the region with rich facial expression features, and the region with relatively insufficient expression feature information is not completely ignored, but is given a certain weight according to the amount of expression feature information it contains, fully capturing the detailed information of the region related to facial expression, and suppressing the feature expression of the region unrelated to facial expression;

[0068] Step 2.2, the RoI feature extraction network is composed of the fourth SE-Res module in the SE-ResNet-50 network, as shown in Figure 1 , and the face region in the face region set Z is taken as the recognition boundary of the face image of the fourth SE-Res module;

[0069] The fourth SE-Res module uses the pooling layer to obtain n=9 RoI feature maps from the global expression information weight feature map P t and performs size unification to obtain the RoI feature map set R t of the t-th face image x t ={R t1 , …, R tk , …, P tn}, wherein R tk represents the k-th size-unified RoI feature map, and n represents the number of RoI feature maps;

[0070] The fourth SE-Res module performs fine-grained feature extraction on the RoI feature map set R t to obtain more detailed RoI feature representation, ensures that the model pays attention to the detailed information, and obtains the high-level semantic feature G t ={G t1 , …, G tk, ..., G tn}, where G tk R represents the feature map after the k-th size is unified. tk The high-level semantic features are obtained, thus yielding the high-level semantic feature sequence G. Q ={G1, ..., G t , ..., G T};

[0071] Step 2.3, the spatiotemporal feature decoupling network, includes, in sequence: L=8 layers of spatial attention modules, L=8 layers of temporal attention modules, a global average pooling layer, and a fully connected layer, as follows: Figure 5 As shown:

[0072] Step 2.3.1, each layer of spatial attention module, such as Figure 6 As shown, it consists of S = 3 attention heads, a linear layer, a normalization layer (BN), and an activation function layer (ReLU). The rounded rectangle on the left represents the attention head.

[0073] High-level semantic feature sequence G Q The input is fed into the first-level spatial attention module, and then fed into G by S=3 attention heads. Q Spatial attention feature map calculation is performed to obtain the high-level semantic features G of S = 3 attention head pairs in frame t. ti Output spatial attention feature map like Figure 7 As shown; thus, the high-level semantic feature sequence G is obtained. Q Spatial attention feature map In the first-layer spatial attention module, the i-th attention head first determines the high-level semantic feature G. t The position of each RoI in G t Spatial location encoding is performed because G t The individual RoIs in the G do not have a predefined order or structure to indicate their identity, therefore a positional encoding is needed to identify them. t Each RoI in the dataset is provided with a unique spatial label, and the spatial location is encoded with the high-level semantic features G of the t-th frame. t The summation, followed by linear function operations, yields the high-level semantic features G of the t-th frame with spatial location encoding for the i-th attention head. ti Thus, the high-level semantic features G of the t-th frame with spatial location coding are calculated using equation (1). ti Spatial attention feature map And using equation (2), the spatial attention feature map A of the i-th attention head of the first-layer spatial attention module is obtained. 1i Attention maps are calculated only for the high-level semantic features in the current video frame, and attention maps are calculated frame by frame. Finally, the attention maps of all video frames are averaged.

[0074]

[0075]

[0076] In equation (1), σ and φ are two embedding functions, (G ti )' represents high-level semantic feature G ti The transpose of ; softmax represents the normalization exponential function, and tanh represents the activation function;

[0077] The spatial attention feature map A output by the i-th attention head of the first-layer spatial attention module. 1i With high-level semantic feature sequence G Q The matrix is ​​obtained after multiplication. This yields spatial attention feature maps of S attention heads, which are then compared with the high-level semantic feature sequence G. Q After multiplying and concatenating the matrices, the spatial attention module contains a total of S attention heads to ensure the model focuses on information from different subspaces. The matrix is ​​then processed sequentially through a linear layer, a normalization layer (BN), and an activation function layer (ReLU) to finally obtain the spatial output matrix.

[0078] Step 2.3.2: Output matrix from the output space Transpose to form a time matrix Y consisting of T = 16 time elements. Q Furthermore, each time element has a matrix of n=9 C channels, where each C=64 channel represents a time set of high-level semantic features. Therefore, the n=9 C=64 channels are regarded as n=9 sets, denoted as Y. Q ={Y1, ..., Y} k , ..., Y n}; where Y k Represents the time set of the k-th high-level semantic features;

[0079] Step 2.3.3: Each temporal attention module consists of S = 3 attention heads, a linear layer, a normalization layer (BN), and an activation function layer (ReLU) in sequence. The calculation of the temporal attention module is similar to that of the spatial attention module.

[0080] Time matrix Y Q The input is fed into the first-level time attention module, and S attention heads are connected to Y. Q Temporal attention feature maps are calculated to obtain the temporal attention feature maps output by S attention heads. Thus, the temporal attention feature map of matrix Y is obtained. In this process, the i-th attention head of the first-layer temporal attention module first processes Y according to the frame order in the video expression sequence Q. k Perform time-location encoding, because Yk Each channel in Y has no predefined order or structure to show its identity, thus a time position encoding is needed to provide Y k Each channel in Y with a unique time mark, and the time position encoding is added to Y k of the i-th attention head of the first layer time attention module, and then the time position encoding is obtained through the operation of the linear function ki , so as to calculate the time set Y of the k-th high-level semantic feature with time position encoding of the video expression sequence Q by formula (3) ki The time attention feature map of the i-th attention head of the first layer time attention module And formula (4) is used to obtain the time attention feature map B 1i :

[0081]

[0082]

[0083] In formula (3), (Y ki )' represents the transpose of Y ki ;

[0084] The time attention feature map B 1i output by the i-th attention head of the first layer time attention module is multiplied by the matrix Y Q to obtain the matrix So as to obtain the matrix of the time attention feature map of the S attention heads after being multiplied by the matrix Y Q , and then sequentially pass through the linear layer, the normalization layer BN, and the activation function layer ReLU, and finally obtain the time output matrix

[0085] Step 2.3.4, convert the time output matrix into a spatial matrix with n=9 spatial elements, each of which has T C channels; according to the process of step 2.3.1-step 2.3.3, sequentially pass the n=9 spatial elements through the next layer of spatial attention module and time attention module, so as to obtain the time output matrix

[0086] Step 2.3.5, sequentially pass the time output matrix through the global average pooling layer and the fully connected layer to obtain a one-dimensional vector a;

[0087] Step 2.4: Input the one-dimensional vector α into the classification module consisting of two fully connected layers and a Softmax function for processing to obtain the label prediction probability of the video expression sequence Q;

[0088] Step 2.5, the label correction module includes: a self-attention weight module, a weight ranking module, and a label re-labeling module, such as... Figure 8 As shown;

[0089] The video facial expression sequence Q is processed by a self-attention weighting module consisting of two fully connected layers and a sigmoid activation function to obtain a weight score a. Q ;

[0090] The weighted ranking module uses a set weight standard value λ, which is a number between 0 and 1. Experiments show that setting λ to 0.7 yields the best results. The weight score a is then... Q Classified into high-weight restructuring or low-weight restructuring;

[0091] If the weight score of the video expression sequence Q is a Q In low-weight recombination, and the maximum prediction probability P in the label prediction probability vector. max The corresponding tag g max If the difference between Q and its true label is greater than the set threshold ξ = 0.2, then Q's label will be relabeled as the maximum predicted probability P. max The tag g corresponding to the rate max Otherwise, preserve the original labels of the video expression sequence Q.

[0092] Step 3: Take the sentiment category corresponding to the maximum value of the label prediction probability of the video expression sequence Q as the final sentiment classification result.

[0093] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor in executing the video expression recognition method, and the processor is configured to execute the program stored in the memory.

[0094] In this embodiment, a computer-readable storage medium stores a computer program, which, when run by a processor, executes the steps of the video expression recognition method.

Claims

1. A video expression recognition method based on spatiotemporal feature decoupling, characterized in that, The procedure is as follows: Step 1: Obtain a video expression dataset with labels in an indoor environment, and perform length standardization on each video in the video expression dataset. Then, standardize the size of the T-frame face images in each standardized video to obtain the preprocessed video expression sequence corresponding to each video. Using the Dlib tool, the face image x in the t-th frame of the preprocessed video expression sequence Q corresponding to any video is obtained. t Facial feature points are marked, and then, based on these facial feature points, facial motion unit segmentation rules are used to segment the face image x in frame t. t The face region in the image is divided to obtain the face image x of frame t. t face region set Z t Thus, the set of facial regions Z of the video expression sequence Q is obtained; Step 2: Construct an expression recognition network, including: a global feature extraction region layer network, a RoI feature extraction network, a spatiotemporal feature decoupling network, a classification module, and a label correction module; Step 2.1: The global feature extraction region layer network consists of the first three modules of the SE-ResNet-50 network and the region layer; The video expression sequence Q is input into the global feature extraction region layer network, and after feature extraction by the first three modules, the feature map sequence f of the video expression sequence Q is obtained. Q ={f1, ..., f t , ..., f T }, where f t x represents the face image in frame t. t The feature map, where T represents the number of frames in the video expression sequence Q; The region layer first processes the t-th frame face image x t Feature map f t After uniformly dividing the image into i×i image blocks, normalization and ReLU activation function operation are performed on each image block to obtain i×i processed image blocks; The region layer uses a C-channel convolutional layer with a 1×1 kernel to perform local convolution operations on each processed image patch, obtaining a feature vector with C channels for each image patch, thus resulting in a set b composed of the feature channel vectors of i×i image patches. t ; The region layer sequentially utilizes two fully connected layers and a sigmoid function to process the feature channel vector set b. t Correlation calculations are performed between feature channels to obtain the weight corresponding to each feature channel. Then, the weight corresponding to each feature channel is multiplied and weighted by the processed i×i image patches to obtain a set a of i×i weighted image patches containing C feature channels. t ; Image patch set a t With feature map f t After matching the positions of the i×i image patches, the feature map f is obtained. t Global facial expression information weight feature map P t Thus, the global facial expression information weight feature map sequence P is obtained. Q ={P1, ..., P t , ..., P T }; Step 2.2: The RoI feature extraction network consists of the fourth SE-Res module in the SE-ResNet-50 network; and the face regions in the face region set Z are used as the recognition boundaries of the face images of the fourth SE-Res module; The fourth SE-Res module utilizes a pooling layer to extract the global facial expression information weight feature map P. t Obtain n RoI feature maps and unify their sizes to obtain the face image x of the t-th frame. t RoI feature map set R t ={R t1 , ..., R tk , ..., P tn }, where R tk This represents the feature map of the k-th RoI after size unification, and n represents the number of RoI feature maps; The fourth SE-Res module processes the RoI feature map set R t Fine-grained feature extraction is performed to obtain high-level semantic features G from n RoI feature maps. t ={G t1 , ..., G tk , ..., G tn }, where G tk R represents the feature map after the k-th size is unified. tk The high-level semantic features are obtained, thus yielding the high-level semantic feature sequence G. Q ={G1, ..., G t , ..., G T }; Step 2.3: The spatiotemporal feature decoupling network comprises, in sequence: an L-layer spatial attention module, an L-layer temporal attention module, a global average pooling layer, and a fully connected layer. Step 2.3.1: Each spatial attention module consists of S attention heads, a linear layer, a normalization layer (BN), and an activation function layer (ReLU). High-level semantic feature sequence G Q The input is fed into the first-layer spatial attention module, and S attention heads are used to focus on G. Q Spatial attention feature map calculation is performed to obtain S attention head pairs for high-level semantic features G in frame t. ti Output spatial attention feature map Thus, the high-level semantic feature sequence G is obtained. Q Spatial attention feature map In the first-layer spatial attention module, the i-th attention head first determines the high-level semantic feature G. t The position of each RoI in G t Perform spatial location encoding, and then combine the spatial location encoding with the high-level semantic features G of frame t. t The summation, followed by linear function operations, yields the high-level semantic features G of the t-th frame with spatial location encoding for the i-th attention head. ti Thus, the high-level semantic features G of the t-th frame with spatial location coding are calculated using equation (1). ti Spatial attention feature map And using equation (2), the spatial attention feature map A of the i-th attention head of the first-layer spatial attention module is obtained. 1i : In equation (1), σ and φ are two embedding functions, (G ti )' represents high-level semantic feature G ti The transpose of ; softmax represents the normalization exponential function, and tanh represents the activation function; The spatial attention feature map A output by the i-th attention head of the first-layer spatial attention module. 1i With high-level semantic feature sequence G Q The matrix is ​​obtained after multiplication. This yields spatial attention feature maps of S attention heads, which are then compared with the high-level semantic feature sequence G. Q The multiplied matrices are concatenated and then processed sequentially through a linear layer, a normalization layer (BN), and an activation function layer (ReLU) to finally obtain the spatial output matrix. Step 2.3.2: Output matrix from the output space Transpose to form a time matrix Y consisting of T time elements Q Furthermore, each time element has a matrix of n C channels, where each C channel represents a time set of high-level semantic features. Thus, the n C channels are treated as n time sets, denoted as Y. Q ={Y1, ..., Y} k , ..., Y n }; where Y k Represents the time set of the k-th high-level semantic features; Step 2.3.3: Each temporal attention module consists of S attention heads, a linear layer, a normalization layer (BN), and an activation function layer (ReLU) in sequence. Time matrix Y Q The input is fed into the first-level time attention module, and S attention heads are connected to Y. Q Temporal attention feature maps are calculated to obtain the temporal attention feature maps output by S attention heads. Thus, the temporal attention feature map of matrix Y is obtained. In this process, the i-th attention head of the first-layer temporal attention module first processes Y according to the frame order in the video expression sequence Q. k Perform time position encoding, and then combine the time position encoding with the Y-axis of the k-th C channel. k After summing and then performing linear function operations, we obtain the temporal set Y of the k-th high-level semantic features with temporal location encoding for the i-th attention head. ki Therefore, Equation (3) is used to calculate the temporal set Y of the k-th high-level semantic features with temporal location encoding of the video expression sequence Q. ki Temporal attention feature map And using equation (4), the temporal attention feature map B of the i-th attention head of the first-layer temporal attention module is obtained. 1i : In equation (3), (Y ki )' represents Y ki transpose; The time attention feature map B output by the i-th attention head of the first-layer time attention module. 1i With matrix Y Q The matrix is ​​obtained after multiplication. Thus, the temporal attention feature maps of S attention heads are obtained and compared with matrix Y. Q The multiplied matrices are concatenated and then processed sequentially through a linear layer, a normalization layer (BN), and an activation function layer (ReLU) to obtain the time output matrix. Step 2.3.4: Output the time matrix The matrix is ​​transformed into a spatial matrix with n spatial elements, each having T C channels. Following steps 2.3.1-2.3.3, the n spatial elements are sequentially processed by the next layer of spatial attention module and temporal attention module, resulting in a temporal output matrix after being processed by L layers of spatial attention modules and L layers of temporal attention modules. Step 2.3.5: Output the time matrix After passing through a global average pooling layer and a fully connected layer in sequence, a one-dimensional vector α is obtained; Step 2.4: Input the one-dimensional vector α into the classification module consisting of two fully connected layers and a Softmax function for processing to obtain the label prediction probability of the video expression sequence Q; Step 2.5: The label correction module includes: a self-attention weight module, a weight ranking module, and a label re-labeling module; The video facial expression sequence Q is processed by the self-attention weighting module to obtain a weight score a. Q ; The weighted ranking module sorts the weight scores a according to the set weight standard value λ. Q Classified into high-weight restructuring or low-weight restructuring; If the weight score of the video expression sequence Q is a Q In low-weight recombination, and the maximum prediction probability P in the label prediction probability vector. max The corresponding tag g max If the difference between Q and its true label is greater than the set threshold ξ, then Q's label will be relabeled as the maximum predicted probability P. max The tag g corresponding to the rate max Otherwise, preserve the original labels of the Q-sequence of the video expressions; Step 3: Take the sentiment category corresponding to the maximum value in the label prediction probability vector of the video expression sequence Q as the final sentiment classification result.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the video expression recognition method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the video expression recognition method according to claim 1.

Images