A method for anger emotion recognition based on video stream

Abstract

The present invention discloses a method for anger emotion recognition based on video stream, which belongs to the field of emotion recognition technology. The present invention obtains a spatiotemporally aligned standardized facial image sequence, a 3D skeleton sequence, and a voice segment by preprocessing the video stream of the visitor; uses a 3D-ResNet34 network and a micro-expression optical flow enhancement algorithm to extract features from the standardized facial image sequence to obtain optical flow facial features; uses a spatiotemporal graph convolutional network to extract features from the 3D skeleton sequence to obtain body movement features; extracts features from the voice segment to obtain voice features; uses a multi-head spatiotemporal cross-attention mechanism and an expanded temporal convolution algorithm to fuse the facial features, body movement features, and voice features, and then outputs the anger emotion recognition result through dynamic weight allocation classification. The present invention significantly improves the accuracy of the system in anger emotion recognition as a whole.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to an anger emotion recognition method based on video stream. Background Art

[0002] In the field of emotion recognition, traditional methods often rely on feature splicing or simple weighted fusion, which results in insufficient exploration of the spatiotemporal correlations between facial, body, and voice modalities. For example, angry expressions are often accompanied by arm movements and sudden changes in voice pitch. Traditional recognition methods struggle to capture these cross-modal spatiotemporal synchronization features, resulting in poor anger recognition accuracy and an inability to cope with emotion recognition in complex scenarios. Summary of the Invention

[0003] The main purpose of the present invention is to provide a method for anger emotion recognition based on video stream, aiming to solve the technical problem that the relevant technology has poor accuracy in anger emotion recognition.

[0004] To achieve the above object, the present invention provides a method for anger emotion recognition based on video stream, the method comprising the following steps:

[0005] S1, preprocess the visitor's video stream to obtain a spatiotemporally aligned standardized facial image sequence, a 3D skeleton sequence, and a voice clip;

[0006] S2, using 3D-ResNet34 network and micro-expression optical flow enhancement algorithm to extract features from standardized facial image sequences and obtain optical flow facial features;

[0007] S3, uses the spatiotemporal graph convolutional network to extract features from the 3D skeleton sequence to obtain limb movement features;

[0008] S4, extracting features from the speech segment to obtain speech features;

[0009] S5, using the multi-head spatiotemporal cross attention mechanism and the dilated temporal convolution algorithm, after fusing the facial features, body movement features and voice features, outputs the anger emotion recognition results through dynamic weight distribution classification.

[0010] The present invention achieves cross-modal spatiotemporal fusion through fine-grained spatiotemporal correlation modeling of facial, body, and voice modalities, thereby fully capturing multimodal collaborative features. At the same time, combined with dynamic weighting of the optical flow field, the system's ability to recognize instantaneous micro-expressions can be enhanced. On this basis, combined with the stability assessment of temporal features and the gated update mechanism, dynamic classification weights are established to achieve continuous quantitative analysis of emotions. Therefore, the present invention significantly improves the accuracy of the system's recognition of anger emotions as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of an embodiment of a method for identifying anger emotions based on video streams according to the present invention;

[0012] Figure 2 The figure is a detailed flow chart of an embodiment of the method for anger emotion recognition based on video stream of the present invention.

[0013] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0014] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0015] The inventive concept of the present application is further described below with reference to some specific embodiments and implementation methods.

[0016] The embodiment of the present invention provides a method for identifying anger emotions based on video streams, referring to Figure 1 , Figure 1 The figure is a flow chart of an embodiment of a method for anger emotion recognition based on video streams according to the present invention.

[0017] In this embodiment, the anger emotion recognition method based on video stream includes:

[0018] Step S1: Preprocess the visitor's video stream to obtain a spatiotemporally aligned standardized facial image sequence, a 3D skeleton sequence, and a voice segment.

[0019] The step S1 specifically includes the following steps:

[0020] Step S11: Divide the video stream into video segments of fixed length, each video segment contains 16 frames of original images, and the overlapping window of each segmentation is 8 frames.

[0021] Anger can be short bursts or continuous, so keeping the segments overlapping can capture more coherent changes and avoid missing key frames.

[0022] Step S12: Extract facial ROI from the original image of each video clip to obtain a standardized facial image sequence .

[0023] Specifically, the RetinaFace detector is used to locate the facial area in the original image, and the key points are aligned through affine transformation to obtain a standardized facial image:

[0024] ;

[0025] in, Represents the original image of the tth frame in the video clip; represents the standardized facial image corresponding to the original image of the t-th frame; Indicates alignment operation; represents the affine transformation parameters; Represents the translation parameter in the x-axis direction; Represents the translation parameter in the y-axis direction; Indicates the rotation angle; Indicates the scaling ratio; Represents the two-dimensional Euclidean group, which is a set of transformations including translation, rotation and scaling.

[0026] After all original images are processed, the standardized facial image sequence of the video clip is obtained .

[0027] In step S12, normalizing the facial images ensures consistent facial position and size across different videos, reducing noise. RetinaFace detects key points, and the aligned facial features (such as eyebrows and mouth changes) are clearer, helping to capture features such as widened eyes and drooped mouth corners in anger.

[0028] Step S13: For the original image of each video clip, use AlphaPose to detect the 2D skeleton joints and generate a 3D skeleton sequence through inverse kinematics optimization .

[0029] Assume that the number of joint points J = 17. The joint points can be used to capture body movements such as arms waving and body leaning forward when a person is angry.

[0030] Step S14: extracting the audio stream from the video stream, performing pre-emphasis filtering, framing (e.g., 25ms window, 10ms overlap) and silent segment filtering in sequence, and retaining 16 frames of voice segments synchronized with the video segments.

[0031] By pre-emphasis filtering and filtering out silent segments, high-energy speech frames that match the characteristics of angry speech bursts can be retained.

[0032] In the entire step S1, the video stream is preprocessed through spatiotemporal segmentation, facial ROI extraction, limb skeleton extraction, and voice processing to achieve multimodal decoupling, thereby systematically extracting and standardizing facial, limb, and voice features, and combining with the temporal alignment mechanism to significantly improve the robustness and accuracy of anger emotion recognition.

[0033] Step S2: Use the 3D-ResNet34 network and micro-expression optical flow enhancement algorithm to extract features from the standardized facial image sequence to obtain optical flow facial features .

[0034] The step S2 specifically includes the following steps:

[0035] Step S21: Normalize the facial image sequence Input 3D-ResNet34 network and use 3D convolution kernel to extract spatiotemporal feature sequence :

[0036] ;

[0037] Among them, the spatiotemporal feature sequence The dimension is 16×512; Represents a 3D convolution operation, which includes four residual blocks connected in sequence. Each residual block consists of a 3D convolution layer, a batch normalization layer, and a ReLU activation layer connected in sequence.

[0038] Step S22: performing micro-expression optical flow enhancement processing on the standardized facial image sequence to obtain an optical flow feature sequence.

[0039] The step S22 specifically includes the following steps:

[0040] Step S22-1: sequentially calculating the dense optical flow fields of adjacent frames in the standardized facial image sequence :

[0041] ;

[0042] in, Represents the FlowNet model; represents the normalized facial image of frame t; Represents the standardized facial image of the t+1th frame, dense optical flow field The dimensions are 112×112×2.

[0043] Step S22-2: Based on dense optical flow field Detect micro-expression moments and obtain micro-expression weight coefficients :

[0044] ;

[0045] in, Indicates the optical flow amplitude; represents the average pooling operation; Represents a learnable convolution kernel with a dimension of 1×1×1, which is used to map the global average pooling result of the optical flow amplitude to a scalar weight; Represents the Sigmoid function.

[0046] Step S22-3: Use 1D convolution kernel to dense optical flow field Perform dimensionality reduction processing to obtain optical flow features; return to step S22-1 until all standardized facial images are processed and the optical flow feature sequence is obtained. :

[0047] ;

[0048] in, Represents the optical flow features corresponding to the standardized facial image of the tth frame; Represents a convolution operation.

[0049] Step S23: Optical flow feature sequence and spatiotemporal feature sequences After splicing frame by frame, use the micro-expression weight coefficient Weighted to obtain optical flow facial features :

[0050] ;

[0051] in, Represents the optical flow facial features corresponding to the standardized facial image of the t-th frame, with a dimension of 567; represents the spatiotemporal features corresponding to the standardized facial image of frame t; Represents a splicing operation.

[0052] Finally, the 16 frames of optical flow facial features Optical flow facial features are obtained after splicing .

[0053] Throughout step S2, the 3D-ResNet34 network and optical flow enhancement technology are combined to efficiently extract dynamic facial features. This not only captures subtle changes that are difficult to detect in static images, but also enhances sensitivity to micro-expressions through optical flow, thereby improving the accuracy and robustness of micro-expression recognition.

[0054] Step S3: Use the spatiotemporal graph convolutional network to extract features from the 3D skeleton sequence to obtain limb movement features .

[0055] The step S3 specifically includes the following steps:

[0056] Step S31: Based on the 3D skeleton sequence, a spatiotemporal graph is constructed through spatial edges and temporal edges.

[0057] Specifically, spatial edges connect adjacent joint points, and temporal edges connect the same joint points in adjacent frames, thus forming a complete spatiotemporal graph.

[0058] Step S32: performing spatial graph convolution on the skeleton features of each time frame in the space-time graph to extract local spatial features.

[0059] Step S33: Perform one-dimensional convolution on the local spatial features along the time dimension to capture the temporal dynamic changes of the action.

[0060] Step S34: Based on the joint attention mechanism, the dot product of query, key and value is calculated for the convolution result and the softmax function is applied to output the limb movement features. .

[0061] In step S3, the spatial-temporal graph simultaneously models both spatial posture and temporal dynamics, capturing the unique motion patterns of anger. Furthermore, the attention mechanism can highlight key joints, reduce noise interference, distinguish subtle motion differences, and improve the accuracy of anger motion recognition.

[0062] Step S4: Extract features from the speech segment to obtain speech features .

[0063] The step S4 specifically includes the following steps:

[0064] Step S41: Calculate and extract 28-dimensional static MFCC parameters based on the speech segment, then obtain 28-dimensional dynamic MFCC features through first-order difference processing, and concatenate the static MFCC parameters and the dynamic MFCC features to obtain a 56-dimensional MFCC feature vector.

[0065] Step S42: Using the SWT-HS algorithm to detect the fundamental frequency of the speech segment, and then calculating the jitter coefficient of the speech segment according to the fundamental frequency:

[0066] ;

[0067] in, Indicates the jitter coefficient; Represents the fundamental frequency value of the t-th frame speech data; Indicates the fundamental frequency value of the t+1th frame of speech data.

[0068] Step S43: Based on the speech segment, calculate the global average spectrum centroid:

[0069] ;

[0070] ;

[0071] in, Represents the centroid of the spectrum of the t-th frame speech data; Represents the discrete Fourier transform coefficient of the kth frequency component of the tth frame of speech data; represents the frequency value of the kth frequency component; represents the global average spectrum centroid.

[0072] Step S44: The speech feature vector is obtained by splicing the MFCC feature vector, the jitter coefficient and the global average spectrum centroid, and then aligned to 16 frames by dynamic time warping to obtain the speech feature vector. .

[0073] In step S4, by combining multiple acoustic features, we can capture the multi-dimensional manifestations of anger in speech, including spectrum, fundamental frequency perturbations, energy, and temporal characteristics, thereby improving the accuracy of anger recognition. Furthermore, time alignment enhances the system's adaptability to different speech patterns, making feature representation more consistent and facilitating the learning of key patterns of anger.

[0074] Step S5: Utilizing the multi-head spatiotemporal cross attention mechanism and the dilated temporal convolution algorithm, the facial features, body movement features, and voice features are fused and processed, and the anger emotion recognition result is output through dynamic weight distribution classification.

[0075] like Figure 2 As shown, step S5 specifically includes the following steps:

[0076] Step S51: facial features , body movement characteristics and voice features Map to the unified space to obtain the facial feature vector , body movement feature vector and speech feature vectors :

[0077] ;

[0078] in, Represents the input features m, namely facial features, body movement features and speech features; The feature vector representing feature m; Represents the weight matrix corresponding to feature m; Represents the bias term corresponding to feature m; Represents the dimension of feature m; Represents a normalization operation.

[0079] Based on the above feature mapping method, the facial feature vector is finally obtained , body movement feature vector and speech feature vectors .

[0080] Step S52: Based on facial feature vector , body movement feature vector and speech feature vectors Perform cross-attention calculations to obtain the attention of facial features to body movement features, the attention of facial features to voice features, the attention of body movement features to facial features, the attention of body movement features to voice features, the attention of voice features to facial features, and the attention of voice features to body movement features:

[0081] ;

[0082] ;

[0083] in, Indicates attention paid to facial features over body movement features; represents the query vector; represents the key vector; represents a value vector; 、 and Both represent linear transformation matrices with a dimension of 512×64; represents the key vector dimension; Represents the key vector The transpose of .

[0084] Similarly, we can obtain the attention of facial features to voice features , attention to facial features by body movement features , attention of body movement features to speech features , attention of voice features to facial features and the attention paid to body movement features by speech features .

[0085] Step S53: Use 4 layers of dilated causal convolution to perform facial feature vector , body movement characteristics , speech feature vector The cross-attention calculation results are fused to obtain facial fusion features, body movement fusion features, and speech fusion features; the expansion rates of the 4 layers of dilated causal convolution are 1, 2, 4, and 8 respectively:

[0086] ;

[0087] in, Indicates facial fusion features; Represents a splicing operation; Represents a dilated causal convolution operation.

[0088] Similarly, the fusion features of body movements can be obtained and speech fusion features .

[0089] Step S54: Calculate the temporal variance of facial features, body movement features, and voice features:

[0090] ;

[0091] in, represents the variance of feature m; Represents the t-th frame feature of feature m; represents the feature mean of feature m.

[0092] Step S55: establishing a dynamic classification weight based on the time series variance to obtain the classification weight at the current moment:

[0093] The step S55 specifically includes the following steps:

[0094] Step S55-1: Setting initial classification weights .

[0095] Step S55-2: Use the gated recurrent unit (GRU) to dynamically update the classification weight to obtain the classification weight at the current moment:

[0096] First calculate the forget gate and update gate :

[0097] ;

[0098] in, Represents the weight matrix of the forget gate, with a dimension of 2×64; Represents the weight matrix of the update gate, with a dimension of 2×64; Represents the classification loss at the previous moment.

[0099] Then calculate the candidate state :

[0100] ;

[0101] in, represents the hyperbolic tangent function, which is used to introduce nonlinearity; Represents the mapping matrix, with a dimension of 2×64, used to map the input signal to the hidden state space; Represents the Hadamard product (element-wise multiplication).

[0102] Finally calculate the classification weight at the current moment :

[0103] ;

[0104] in, Represents the classification weight at the previous moment.

[0105] Step S56: Based on the classification weight at the current moment, weighted aggregation of facial fusion features, body movement fusion features, and voice fusion features is performed, and the classification probability is output to obtain the anger emotion recognition result:

[0106] ;

[0107] in, represents the classification probability, including two emotion categories (angry / not angry); Represents the fusion feature of feature m.

[0108] Finally, the emotion category corresponding to the maximum probability value is taken as the anger emotion recognition result.

[0109] Throughout step S5, unified feature space projection and a multimodal cross-attention mechanism dynamically integrate the spatiotemporal features of the face, body language, and voice, effectively capturing discriminative information about the multimodal co-evolution of anger (such as the synchronization of facial contortions and body movements). Furthermore, dilated convolutions are used to model long-term dependencies to locate emotional outbursts. Combined with GRU dynamic weight allocation, the algorithm adaptively balances modal contributions under noise interference, achieving highly robust anger recognition in complex scenarios (including occlusion and noise), significantly improving the accuracy of detecting micro-expressions and rapid emotional changes.

[0110] In this embodiment, through fine-grained spatiotemporal correlation modeling of facial, body, and voice modalities, cross-modal spatiotemporal cross-fusion is achieved, thereby fully capturing multimodal collaborative features. At the same time, combined with dynamic weighting of the optical flow field, the system's ability to recognize instantaneous micro-expressions can be enhanced. On this basis, combined with the stability assessment of temporal features and the gated update mechanism, dynamic classification weights are established to achieve continuous quantitative analysis of emotions. Therefore, the present invention significantly improves the accuracy of the system's recognition of anger emotions as a whole.

[0111] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0112] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for anger emotion recognition based on video stream, characterized in that: The method comprises the following steps: S1, preprocess the visitor's video stream to obtain a spatiotemporally aligned standardized facial image sequence, a 3D skeleton sequence, and a voice clip; S2, using 3D-ResNet34 network and micro-expression optical flow enhancement algorithm to extract features from standardized facial image sequences and obtain optical flow facial features; The S2 specifically includes: S21, normalizes the facial image sequence Input 3D-ResNet34 network and use 3D convolution kernel to extract spatiotemporal feature sequence ; S22, performing micro-expression optical flow enhancement processing on the standardized facial image sequence to obtain an optical flow feature sequence ; The S22 specifically includes: S22-1, sequentially calculating the dense optical flow fields of adjacent frames in the standardized facial image sequence : ； in, Represents the FlowNet model; represents the normalized facial image of frame t; Represents the standardized facial image of the t+1th frame, dense optical flow field The dimensions are 112×112×2; S22-2, based on dense optical flow field Detect micro-expression moments and obtain micro-expression weight coefficients : ； in, Indicates the optical flow amplitude; represents the average pooling operation; Represents a learnable convolution kernel with a dimension of 1×1×1, which is used to map the global average pooling result of the optical flow amplitude to a scalar weight; Represents the Sigmoid function; S22-3, using 1D convolution kernel for dense optical flow field Perform dimensionality reduction processing to obtain optical flow features; return to step S22-1 until all standardized facial images are processed and the optical flow feature sequence is obtained. : ； in, Represents the optical flow features corresponding to the standardized facial image of the tth frame; Represents the convolution operation; S23, the optical flow feature sequence and spatiotemporal feature sequences After splicing frame by frame, use the micro-expression weight coefficient Weighted to obtain optical flow facial features ; S3, uses the spatiotemporal graph convolutional network to extract features from the 3D skeleton sequence to obtain limb movement features; S4, extracting features from the speech segment to obtain speech features; S5, using the multi-head spatiotemporal cross attention mechanism and the dilated temporal convolution algorithm, after fusing the facial features, body movement features and voice features, outputs the anger emotion recognition results through dynamic weight distribution classification.

2. The anger emotion recognition method based on video stream according to claim 1, characterized in that Said S1 specifically includes: S11, split the video stream into video segments of fixed length, each video segment contains 16 frames of original images, and the overlapping window of each segmentation is 8 frames; S12, performing facial ROI extraction on the original image of each video clip to obtain a standardized facial image sequence of the video clip; S13, for the original image of each video clip, AlphaPose is used to detect the 2D skeleton joints, and a 3D skeleton sequence is generated through inverse kinematics optimization; S14, extracting the audio stream from the video stream, performing pre-emphasis filtering processing, frame processing and silent segment filtering processing in sequence, and retaining 16 frames of voice segments synchronized with the video segments.

3. The anger emotion recognition method based on video stream according to claim 1, characterized in that The S3 specifically includes: S31, based on the 3D skeleton sequence, constructs a spatiotemporal graph through spatial edges and temporal edges; S32, performing spatial graph convolution on the skeleton features of each time frame in the space-time graph to extract local spatial features; S33, performs one-dimensional convolution on the local spatial features along the time dimension to capture the temporal dynamic changes of the action; S34, based on the joint attention mechanism, calculates the dot product of query, key and value on the convolution result and applies the softmax function to output the limb movement features. .

4. The method for anger emotion recognition based on video stream according to claim 1, wherein: The S4 specifically includes: S41, extracting 28-dimensional static MFCC parameters based on the speech segment, obtaining 28-dimensional dynamic MFCC features through first-order difference processing, and concatenating the static MFCC parameters and the dynamic MFCC features to obtain a 56-dimensional MFCC feature vector; S42: Detect the fundamental frequency of the speech segment using the SWT-HS algorithm, and then calculate the jitter coefficient of the speech segment based on the fundamental frequency: ； in, Indicates the jitter coefficient; Represents the fundamental frequency value of the t-th frame speech data; Indicates the fundamental frequency value of the t+1th frame of speech data; S43, based on the speech segment, calculate the global average spectrum centroid: ； ； in, Represents the centroid of the spectrum of the t-th frame speech data; Represents the discrete Fourier transform coefficient of the kth frequency component of the tth frame of speech data; represents the frequency value of the kth frequency component; represents the global average spectrum centroid; S44, based on the MFCC feature vector, jitter coefficient and global average spectrum centroid splicing to obtain the speech feature vector, and then align it to 16 frames through dynamic time warping to obtain the speech feature .

5. The method for anger emotion recognition based on video stream according to claim 1, wherein: The S5 specifically includes: S51, facial features , body movement characteristics and voice features Map to the unified space to obtain the facial feature vector , body movement feature vector and speech feature vectors ; S52, based on facial feature vector , body movement feature vector and speech feature vectors Perform cross-attention calculations to obtain the attention of facial features to body movement features, the attention of facial features to voice features, the attention of body movement features to facial features, the attention of body movement features to voice features, the attention of voice features to facial features, and the attention of voice features to body movement features; S53, using 4 layers of dilated causal convolution, for facial feature vectors , body movement characteristics , speech feature vector The cross-attention calculation results are fused to obtain facial fusion features, body movement fusion features, and speech fusion features; the expansion rates of the 4 layers of dilated causal convolution are 1, 2, 4, and 8 respectively; S54, calculating the temporal variance of facial features, body movement features, and voice features; S55, establishing a dynamic classification weight based on the time series variance to obtain the classification weight at the current moment; S56, based on the classification weight at the current moment, weightedly aggregate the facial fusion features, the body movement fusion features, and the voice fusion features, output the classification probability, and obtain the anger emotion recognition result.

6. The method for anger emotion recognition based on video stream according to claim 5, characterized in that: The S55 specifically includes: S55-1, set initial classification weights; S55-2, using the gated recurrent unit to dynamically update the classification weight to obtain the classification weight at the current moment.

Images