Emotion recognition method based on body motion spatiotemporal graph transformer

Abstract

The application discloses a method for emotion recognition based on a body motion space-time graph transformer, and the implementation process of the method is as follows: 1) an input video is acquired and frame-by-frame skeleton key point detection is performed; 2) a human skeleton graph G is constructed according to a joint topological relationship; 3) space joint coding is realized by adopting an adjacency propagation + multi-layer mapping mode; 4) time position coding is added to form a space-time representation sequence; 5) the representation sequence is input into a graph structure transformer, and multi-layer self-attention modeling is performed; and 6) an emotion recognition result is output through a classification head. The method realizes high-precision action-driven emotion recognition by jointly utilizing the space structure of a skeleton, the time dynamic characteristics of an action and the global relationship modeling capability of a transformer. The application introduces a graph perception transformer, adaptively constructs space-time dependent relationships, significantly improves the cross-frame global modeling capability and enhances the robustness in a complex scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically to an emotion computing and recognition method and system based on human dynamic behavior, and more particularly to an emotion recognition system based on a spatiotemporal graph transformer architecture, which identifies emotional states by analyzing the dynamic changes of human skeleton sequences, and is suitable for scenarios where facial information is limited or unreliable. Background Technology

[0002] Understanding human emotions is central to building intelligent and empathetic computing systems. Over the past decade, affective computing has primarily focused on facial expression analysis, inferring emotional states through visual data and convolutional neural networks (CNNs). Facial images, facial landmarks, and facial action units (FAUs) have become key cues for identifying emotional information. While these technologies have achieved significant success, their reliability drops dramatically in real-world scenarios when faces are partially occluded, blurred, or completely invisible. More importantly, facial expressions alone cannot capture the full picture of human emotional expression, especially in dynamic or socially rich contexts.

[0003] In contrast, the emotional information conveyed by human body postures, gestures, and subtle movements constitutes a rich yet underutilized channel for emotional communication. Despite its growing importance, emotion recognition based on body posture remains a relatively unexplored area. Theoretically, this research breaks through the traditional boundaries between action recognition and emotion computing. Although actions and emotions are often considered independent categories, we argue that the essence of emotion is embedded in movement patterns: the amplitude of hand tremors, the tension in the shoulders, and the rhythm of nodding all carry rich emotional signals. This prompts the adoption of graph modeling methods, treating body joints as interconnected entities rather than isolated coordinates. By integrating graph theory with sequence modeling of skeletons, a novel emotion perception architecture is constructed, exhibiting strong robustness, flexibility, and generalization ability in scenarios where facial data is unreliable or missing.

[0004] The existing technology is as follows:

[0005] 1. Emotion recognition based on facial expressions

[0006] Facial expressions, due to their direct association with emotional states, have become the most commonly used modality in visual emotion recognition. They can capture cues at the level of appearance features, enabling fine-grained tracking of expression changes. Facial images, facial landmarks, and facial action units (FAUs) have become key indicators for recognizing emotional information.

[0007] 2. Skeleton sequence modeling of graph convolutional neural networks

[0008] Graph models have proven to have significant advantages in handling the non-Euclidean structures of the human skeleton, naturally encoding joint spatial relationships and extending to temporal dynamic modeling. Enhancement techniques such as Two-Stream Graph Convolutional Networks (2s-GCN) and semantically aware partitioning can improve the ability to model complex motions of emotionally salient body parts. Recent advances explore dynamic graph construction and adaptive connection mechanisms, enabling models to respond to diverse emotional cues.

[0009] The shortcomings of existing technology are:

[0010] 1. Emotion recognition based on facial expressions;

[0011] Facial expressions, due to their direct association with emotional states, have become the most commonly used modality in visual emotion recognition. While they can capture cues at the level of physical features and achieve fine-grained tracking of expression changes, in real-world scenarios, the reliability of relying solely on facial cues significantly decreases when facial expressions are subtle, blurred, occluded, or influenced by cultural background.

[0012] 2. Skeleton sequence modeling for graph convolutional neural networks;

[0013] Existing graph convolutional network models and other techniques that rely on local neighborhood aggregation still exhibit locality and lack global consideration. Furthermore, while these techniques have proven effective in action recognition, they have not yet been applied to sentiment computing. Summary of the Invention

[0014] The technical objective of this invention is to design an emotion recognition method based on a body motion spatiotemporal graph transformer, which mainly achieves:

[0015] 1. Body movements presented in dynamic posture diagrams become the core signal for emotional understanding;

[0016] This invention aims to encourage increasing attention to the integration of complementary modalities such as body posture and speech to achieve more robust emotion recognition. It reconstructs emotion recognition as a structure-aware graph-based temporal reasoning task, where body movement, presented as a dynamic posture graph, becomes the core signal for emotion understanding.

[0017] 2. Construct a global interactive spatiotemporal graph transformer;

[0018] To achieve this goal, an architecture based on a spatiotemporal graph transformer is designed to model spatiotemporal pose data as a graph-structured labeled sequence. Unlike existing graph convolutional network models that rely on local neighborhood aggregation, this method implements a global self-attention mechanism on the learned pose representation, enabling the model to capture long-range interactions and subtle temporal emotional cues.

[0019] The technical solution adopted in this invention is an emotion recognition method and system based on space-time joint coding and graph structure transformers, used to extract emotional states from human action sequences. This method achieves joint modeling of global temporal relationships and skeletal structure information by constructing a human skeletal graph structure, designing a space-time joint coding method, and introducing a graph-aware transformer backbone network.

[0020] The implementation process of this invention is as follows: 1) Acquire the input video and perform frame-by-frame skeletal keypoint detection; 2) Construct a human skeleton graph G based on joint topological relationships; 3) Implement spatial joint encoding using adjacency propagation + multi-layer mapping; 4) Add temporal position encoding to form a spatial-temporal representation sequence; 5) Input the representation sequence into a graph structure transformer for multi-layer self-attention modeling; 6) Output the emotion recognition result through a classification head. This method achieves high-precision action-driven emotion recognition by jointly utilizing the spatial structure of the skeleton, the temporal dynamic features of the action, and the global relationship modeling capability of the transformer.

[0021] The relationship between the above six steps is as follows:

[0022] The entire process is an end-to-end deep learning processing pipeline. The output of each step directly serves as the input for the next step, and the data format is transformed sequentially: from raw pixels (video frames) -> structured data (joint coordinates) -> graph structure data (skeleton diagram) -> higher-order features (spatial encoding) -> spatiotemporal sequence (spatiotemporal representation) -> context-aware features (output of self-attention transformer) -> final prediction (emotion category).

[0023] The emotion recognition method based on body motion spatiotemporal graph transformer includes the following steps:

[0024] Step S1: Obtain the input video sequence V={I1,I2,...I... t ,...I T}, where T is the total number of video frames, and a pre-trained two-dimensional pose estimation model is used for each frame image I. t Perform human skeleton keypoint detection to obtain the joint point coordinate sequence P={J1,J2,...J} for each frame. t ,...J T}, where J t Let be the coordinate matrix of the K key points in the t-th frame, where each key point contains two-dimensional spatial coordinates (x, y) and detection confidence c;

[0025] Step S2: Construct an undirected graph G for each frame based on the joint topology defined by human physiological structure. t =(V t E t ), where the node set V tFor each of the K key points, the node features are initialized as (x, y, c), and the edge set E t Based on the definition of physical connections between joints, a corresponding adjacency matrix A is constructed to characterize the spatial adjacency relationships between joints.

[0026] Step S3: The skeletal diagram sequence {G1,G2,...,G...} constructed in step S2 is then processed. T The node and its corresponding adjacency matrix A are input into a graph neural network. By stacking multiple graph convolutional layers or graph attention layers, neighborhood information is aggregated along the edges defined by the adjacency matrix. This allows the features of each node to gradually fuse with the spatial structure information of its multi-hop neighbor nodes, generating a joint feature representation Z that integrates local and global spatial structures. t ∈ N×d_model Where d_model is the feature embedding dimension;

[0027] Step S4: Calculate the spatial coding features Z of each frame output from step S3. t Flattened into a vector vec(Z) t )∈ D Where D = K × d_model, and is associated with the learnable temporal location encoding vector p of the corresponding frame. t ∈ D Adding them together yields a spatiotemporal joint representation vector z that integrates spatial neighborhood aggregation features and temporal location information. t =vec(Z t )+p t This leads to the construction of a spatiotemporal token sequence X=[z1;z2;...;z T ]∈ T×D ;

[0028] Step S5: Input the spatiotemporal token sequence X constructed in Step S4 into a graph structure transformer encoder consisting of L stacked layers. Each layer contains a multi-head self-attention module and a feedforward neural network module. The attention weights between any two tokens are calculated through the multi-head self-attention mechanism to dynamically construct global dependencies across frames and joints, forming an adaptive spatiotemporal graph structure and outputting high-level spatiotemporal semantic features. ;

[0029] Step S6: Calculate the high-level spatiotemporal semantic features H output from step S5. L The data is input to a classification head, which is a multilayer perceptron. First, the features are globally pooled or specific frame features are extracted. Then, they are mapped to the sentiment category space through a fully connected layer. Finally, the sentiment category probability distribution is output through a softmax function to complete sentiment recognition.

[0030] Furthermore, the pose estimation model mentioned in step S1 can be any one of YOLO-Pose, OpenPose, HRNet, or MediaPipe. During detection, the human body with the highest confidence in the image is selected as the target object, and the coordinates and confidence information of its K joints are extracted.

[0031] Furthermore, the joint topological relationships described in step S2 include, but are not limited to, the following connections: head-neck, neck-shoulder, shoulder-elbow, elbow-wrist, hip-knee, knee-ankle, and the connection between the trunk center and the root nodes of the limbs. The adjacency matrix A is a symmetric matrix, and self-loops are introduced in the graph convolution operation, that is, each node is connected to itself.

[0032] Further, the graph neural network mentioned in step S3 is any one or a combination of a graph convolutional network, a graph attention network, or a graph transformer; when a graph convolutional network is used, the node feature update formula is:

[0033] ;

[0034] Where A is the adjacency matrix, H (l) Let W be the feature matrix of the nodes in the l-th layer. (l) Let σ be the learnable weight matrix, and σ be the non-linear activation function.

[0035] Furthermore, the graph neural network in step S3 has 2 to 4 stacked layers, and the output feature dimension d_model of each layer ranges from 128 to 1024. Through multi-layer mapping, the feature abstraction from local joints to body parts is realized step by step.

[0036] Furthermore, the learnable temporal position encoding vector p_t mentioned in step S4 is a parameter vector that is randomly initialized and optimized together with the network parameters during model training, and is used to encode inter-frame temporal sequence information; or in another embodiment, a fixed position encoding of sine and cosine functions is used instead of the learnable encoding.

[0037] Furthermore, the calculation process for each layer of the graph structure converter encoder in step S5 includes:

[0038] Sub-step S5-1: For input H (l-1) After layer normalization, the input is given to the multi-head self-attention module, and the attention output is calculated:

[0039] H' (l) = MSA(LN(H (l-1) )) + H (l-1) )

[0040] Where MSA is a multi-head self-attention mechanism, LN is layer normalization, and + indicates residual connection;

[0041] Step S5-2: For H' (l) After layer normalization, the input is fed into the feedforward neural network module to calculate the final output:

[0042] H (l) = FFN(LN(H' (l) )) + H' (l)

[0043] FFN is a feedforward network consisting of two fully connected layers and an activation function.

[0044] The calculation formula for the multi-head self-attention mechanism is as follows:

[0045] Where Q, K, and V are obtained from the input features through linear transformation, and d k The dimension of the attention head is defined by concatenating the outputs of multiple heads and then performing a linear transformation to obtain the final attention output.

[0046] In step S5, the number of stacked layers L of the graph structure transformer encoder is 4 to 12, the number of attention heads per layer is 4 to 16, the dimension of the hidden layer is 256 to 1024, and a dropout rate of 0.1 to 0.5 is used to prevent overfitting.

[0047] Furthermore, in step S6, the classification head first performs global average pooling on H^L or takes the features of specific frames in the sequence as video-level global features, and then maps them to the emotion category space through a multilayer perceptron containing 1 to 3 fully connected layers. The number of neurons in the output layer is equal to the number of emotion categories C, and the softmax function is used to convert the output into a probability distribution.

[0048] The emotional category C includes at least four of the following: anger, neutrality, happiness, and sadness, or may be extended to include more basic emotional categories such as fear, surprise, and disgust.

[0049] Furthermore, the method employs the cross-entropy loss function as the optimization objective during training:

[0050] Where y c One-hot encoding for the real label, ŷ c Let C be the probability of the c-th category predicted by the model, where C is the total number of sentiment categories.

[0051] Furthermore, an emotion recognition system based on a body motion spatiotemporal graph transformer is characterized by comprising:

[0052] The skeletal keypoint detection module is used to extract the coordinates and confidence scores of human joint points in each frame of the input video.

[0053] The skeleton graph construction module is used to construct the graph structure of each frame and generate an adjacency matrix based on the physical connection relationship of the joints.

[0054] The spatial coding module is used to aggregate neighborhood information from the skeleton map through a graph neural network to generate joint feature representations that fuse spatial structures.

[0055] The spatiotemporal coding module is used to fuse spatial features with learnable temporal location codes to construct a spatiotemporal joint representation sequence;

[0056] The graph structure transformer module is used to perform global dependency modeling on spatiotemporal representation sequences through a multi-head self-attention mechanism and output high-level semantic features.

[0057] The classification module maps high-level semantic features to sentiment categories and outputs the recognition results.

[0058] The graph structure transformer module adopts a multi-head self-attention mechanism, which does not rely on a fixed adjacency matrix. Instead, it dynamically calculates the attention weight between any two tokens based on the input features to form an adaptive spatiotemporal graph structure, thereby enabling the modeling of long-range dependencies across frames and joints.

[0059] The spatial coding module employs graph convolutional networks or graph attention networks, and achieves hierarchical feature abstraction from local joints to whole-body posture by stacking multiple layers of networks.

[0060] Furthermore, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the emotion recognition method.

[0061] Compared with existing technologies, the space-time graph transformer emotion recognition method provided by this invention has the following advantages over existing graph convolutional network methods that mainly rely on facial expression features or are based on fixed skeletal neighborhood modeling:

[0062] 1. Beneficial effects from a technical perspective:

[0063] (1) To achieve joint modeling of spatial structure and temporal dynamics, thereby improving the ability to express features.

[0064] This invention constructs a human skeletal graph structure and encodes each frame of skeletal data into a representation that integrates spatial neighborhood aggregation features and learnable temporal location information. This enables the model to simultaneously capture the temporal variation features of joint topology and action sequences. Compared to traditional methods that only utilize spatial or temporal features, this invention obtains a more complete and stable representation of emotion-related action features.

[0065] (2) Introduce a graph-aware transformer to adaptively construct spatiotemporal dependencies.

[0066] This invention overcomes the limitations of traditional GCN, which relies on a fixed adjacency matrix and is difficult to express long-range dependencies. The transformer's self-attention mechanism can dynamically calculate the dependencies between joints based on the input skeletal sequence, forming an adaptive spatiotemporal graph structure that is not limited to physical connections, thereby enhancing the ability to capture emotional and action patterns across limbs and time frames.

[0067] (3) Significantly improves cross-frame global modeling capabilities and enhances robustness in complex scenarios.

[0068] This invention utilizes multi-head self-attention to establish global feature interactions, avoiding the gradient decay and local feature loss problems that occur in traditional graph convolutional networks in long sequences or complex actions, enabling the model to maintain high recognition performance even under conditions of occlusion, irregular action amplitude, and a lot of skeletal noise.

[0069] 2. Beneficial effects at the application level

[0070] (1) Applicable to privacy-sensitive scenarios, improving the security and acceptability of emotion recognition.

[0071] This invention uses only human skeletal coordinate data for emotion recognition, without relying on sensitive identity information such as facial images, and can be securely deployed in scenarios involving privacy protection, such as education monitoring and driving behavior analysis.

[0072] (2) It is suitable for complex and uncontrolled real environments, enabling a wider range of applications.

[0073] The spatiotemporal converter structure of this invention can maintain stable performance even in scenarios with noisy skeletons, drastic posture changes, and partial limb occlusion, making it suitable for application in "natural environments," including complex real-world environments such as online classrooms and human-computer interaction in public spaces. Attached Figure Description

[0074] Figure 1 A diagram of the human skeletal joint structure. In the diagram: each frame of data captures the dynamic changes of the limbs through the three-dimensional coordinates of 16 joints. This representation method, while protecting privacy, provides rich dynamic feature information for emotion measurement. Detailed Implementation

[0075] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0076] This invention utilizes dynamic skeleton change sequences for emotional state recognition;

[0077] This invention proposes an emotion recognition method based on the dynamic change sequence of key points in the human skeleton. By modeling multi-joint coordinate sequences extracted from continuous video frames, it captures dynamic patterns related to emotions, such as human movement rhythm, posture changes, and cross-joint coordinated movements. This method does not rely on facial information or appearance texture, thus achieving high privacy, low sensitivity, and cross-scene emotion inference. The core points protected by this invention include: constructing multi-joint temporal features for each frame of human skeleton data; continuous modeling of the spatial topological relationships between joints; extraction of temporal movement change trends; and the technical approach of using the skeleton sequence itself as the main input for emotion recognition. Its technical scope covers: the technical solution of using the sequence of key points in the human skeleton as input for emotion recognition, extracting cross-frame dynamic features, and realizing emotion recognition.

[0078] This invention presents a skeleton-based emotion recognition framework based on a space-time graph transformer;

[0079] This invention constructs a spatiotemporal graph framework for performing self-attention modeling of graph transformers that jointly integrates spatial structure and temporal dynamics of skeletal sequences. It includes: a spatial graph structure built based on human skeletal topology; a spatial encoding method that propagates multi-joint spatial neighborhood features along graph edges; a temporal representation method combining learnable temporal position embeddings; and a graph-aware multi-head self-attention mechanism to learn global dependencies across joints and across time. This framework can dynamically construct spatiotemporal relational graphs that adapt to changes in action and emotion, overcoming the limitation of traditional graph convolutional networks that rely only on fixed local connections. The technical points protected by this invention include: constructing a representation encoding method based on a skeleton graph, a transformer self-attention mechanism combining graph structures, and a complete spatiotemporal graph transformer model architecture using the above modules to implement emotion classification.

[0080] The specific details of the implementation are as follows:

[0081] Step 1: Acquire the input video and perform frame-by-frame skeletal keypoint detection.

[0082] Input: Original video sequence V={I1,I2,...I... t ,...I T}, where I t Let t be the image of the t-th frame, where t ranges from 1 to T.

[0083] Core task: To transform pixel information into structured human pose data.

[0084] Key implementation details:

[0085] Model Selection: The readily available, high-performance 2D pose estimation model YOLO-Pose is used to detect joints in each video frame and obtain their coordinate values. Detection Process: Each frame's I... tAn independent input pose estimation model is used. The model outputs the two-dimensional coordinates (x, y) of K key points for each human body in the frame and the confidence score c of the key point detection. This invention focuses on emotion recognition and single-person detection; therefore, the human body with the highest confidence score in the image is selected for key point detection.

[0086] The output connects to step 2: The output of step 1 is a serialized data structure: P={J1,J2,...,J...} T}. Where J t P is a (K,3) matrix (or tensor) representing the K joint information of frame t, with each row containing (x coordinate, y coordinate, confidence level c). This P is the direct input for step 2.

[0087] Step 2: Construct the human skeleton map G based on the joint topology.

[0088] Input: The joint sequence P obtained from step 1. Note that at this point, each joint only contains position information and confidence; the relationships between frames and between joints have not yet been explicitly modeled.

[0089] Core task: Define the spatial topology between joints for each frame, represent the joints of the human body as a graph structure, and lay the foundation for subsequent processing based on graph spatiotemporal transformers.

[0090] Key implementation details:

[0091] Definition of a graph: Each frame of the human body is defined as an undirected graph G. t =(V t E t ).

[0092] Node set V t Each joint is a node. Node feature v i Initialize the joint to its (x, y,) coordinates and confidence level c.

[0093] Edge set E t Based on the definition of human physiological structure, common topological connections include: head-neck-shoulder-elbow-wrist, hip-knee-ankle, etc. Each edge represents a physical connection between two joints.

[0094] Adjacency Matrix: In code implementation, graph structures are typically represented using an adjacency matrix A. A is a KxK matrix; if joints i and j are connected, then A is an adjacency matrix. ij =1, otherwise 0.

[0095] The output connects to step 3: The output of step 2 is a graph structure sequence {G1, G2, ..., G...} that defines spatial relationships. T} and its corresponding adjacency matrix A. This graph sequence and adjacency matrix are the basis for spatial encoding in step 3.

[0096] Step 3: Implement spatial joint encoding using adjacency propagation and multi-layer mapping.

[0097] Input: The skeletal graph sequence {G} constructed in step 2 t} and adjacency matrix A.

[0098] Core task: Using the idea of ​​Graph Attention Network (GAT), aggregate the neighbor information of each joint to learn spatial features that can represent local body parts (such as arms and legs) and global pose.

[0099] Key implementation details:

[0100] Adjacency propagation (graph convolution operation): This is the core. For the l-th layer graph convolution, the node feature update formula can be simplified to:

[0101]

[0102] Among them, H (1) Let A be the feature matrix of all nodes in layer l, A be the adjacency matrix (which may contain self-loops), D be the degree matrix (used for normalization), and W be the number of nodes in layer l. (l) σ is a learnable weight matrix, and σ is a non-linear activation function. This operation allows the features of each node to incorporate information from its first-order neighbors.

[0103] Multi-layer mapping: By stacking multiple layers of graph convolutional layers (such as 2-3 layers), information can be propagated from local joints to more distant joints. For example, after the first layer, the "wrist" node incorporates information from the "elbow"; after the second layer, the "wrist" node can indirectly incorporate information from the "shoulder".

[0104] Feature enhancement: The initial node features (x, y) may be too simple. Typically, they are mapped to a higher-dimensional feature space d_model through a learnable linear layer or MLP before being fed into a graph convolutional layer.

[0105] Output: After feature enhancement, the output feature Z = Z t ,t=1…T, where Z t ∈ N×d_model The format is as follows: T represents the number of time frames, N represents the number of joints extracted from each frame, and d_model represents the dimension of the feature vector for each joint. This is the output of the spatial encoding module and is the key data structure connecting the skeleton graph construction and the spatiotemporal transformer.

[0106] Step 4: Spatial-Temporal Joint Coding Module

[0107] Input: The spatially encoded feature sequence Z output from step 3 t ∈ N×d_model , is the spatial coding feature matrix of all joints in frame t, where N is the number of joints and d_model is the feature dimension.

[0108] Core task: To extend spatial features into spatiotemporal joint coding features, enabling the model to perceive the dynamic evolution process over time.

[0109] Key implementation details:

[0110] 1. Time-location encoding injection

[0111] To preserve the order information between frames, a learnable position vector p is introduced for each time frame. t , and p t ∈ D Where D = N × d_model. It can be seen that its dimension is the same as the dimension of the flattened single-frame spatial features. t It is a learnable parameter vector specifically designed to represent the unique position of time point t in the sequence. During training, the model automatically learns the importance of different time positions in emotional expression.

[0112] 2. Constructing a spatiotemporal token sequence

[0113] The spatial coding features of each frame are added to the corresponding temporal location coding to form a token sequence that the transformer can process:

[0114] z t= vec(Z t )+p t

[0115] X=[z1;z2;...;z T ]∈ T×D

[0116] Among them, vec(Z) t ) is the vector whose flattened spatial pose structure of the human body in frame t is (D,); z t It is the complete token representation of the t-th frame. It contains both the spatial structure information vec(Z_t) and the temporal location information p of that frame. t X is the input sequence of the entire video. It is a T×D matrix, which can be regarded as a sequence of length T and feature dimension D for each frame, ready to be input to the transform encoder.

[0117] Output: Spatiotemporal joint token sequence X∈ T×D The sequence length is T.

[0118] Step 5: Graph Structure Transformer (Multi-head Attention Spatiotemporal Modeling)

[0119] Input: The spatiotemporal token sequence X ∈ (output from step 4) T×D Where T is the number of frames and D is the feature dimension.

[0120] Core task: To model the global dependencies of spatiotemporal token sequences using a self-attention transformer encoder, allowing any two tokens in the sequence (i.e., any joint in any frame) to interact directly, thereby capturing spatiotemporal patterns and helping to understand the emotional categories expressed across multiple joints and time.

[0121] Key implementation details:

[0122] 1. Converter Structure

[0123] Let the input of the l-th layer be H l-1 The calculation process of a single-layer converter consists of two flows: "normalization-attention-residual" and "normalization-feedforward-residual". The self-attention module is: H' l =MSA(LN(H l-1 ))+H l-1 LN represents normalization, and self-attention MSA is defined as follows: , + indicates residual connection.

[0124] Feedforward Network (FFN) module is defined as FFN(x) is a nonlinear transformation and activation function that enhances the model's ability to express complex emotions.

[0125] 2. Deep modeling achieved through multi-layer stacking

[0126] The MSA and FFN modules mentioned above are stacked in L layers, where the lower layer captures basic, local spatiotemporal action units; the middle layer combines local units to form regional emotional representations; and the upper layer integrates all spatiotemporal information to establish a global emotional state representation.

[0127] Output: The encoded high-level spatiotemporal representation, i.e.: , which serves as a global summary feature for the entire video.

[0128] Step 6: Output emotion recognition results through the classification head.

[0129] Input: Video-level global features extracted in step 5 .

[0130] Core task: Map spatiotemporal semantic features to a discrete sentiment category space, output the final sentiment category using a classification head, and complete the mapping from the original video to sentiment category labels.

[0131] Key implementation details:

[0132] 1. Classification Header Structure

[0133] The classification head is a multilayer perceptron (MLP) that maps high-dimensional features to a set of sentiment categories C, specifically expressed as follows: .

[0134] 2. Loss Function

[0135] The entire model is trained using the cross-entropy loss function: Where C represents the total number of categories, The value (0 or 1) of class c in the one-hot encoding representing the real label. The value represents the probability of class c predicted by the model (output via softmax), and log refers to the natural logarithm.

[0136] The significance of the loss function: When the loss function makes the probability distribution predicted by the model... As close as possible to the true distribution When the time is right, the loss value is minimized.

[0137] 3. Output and Reasoning

[0138] During the reasoning phase, the category with the highest probability is selected as the predicted sentiment category.

[0139] Output: The sentiment category label predicted from the given input video.

[0140] Simulation

[0141] Dataset

[0142] This invention evaluates the proposed method on the Emotion-Gait dataset (E-Gait), which is a benchmark set closely resembling real-world scenarios, built upon and further optimized by Bhattacharya et al. The dataset contains 2,177 gait sequences, including 342 newly acquired real-world gait records and 1,835 samples sourced from the Edinburgh Motion Capture Database. Each motion frame contains 3D coordinate data for 16 joints (e.g., ...). Figure 1 Furthermore, all skeletal models underwent normalization to eliminate individual variability. The dataset annotations cover four emotion categories: Angry, Neutral, Happy, and Sad.

[0143] Implementation details

[0144] This invention uses Python 3.7 and PyTorch 1.13 to conduct simulation experiments on the proposed method. It includes six spatiotemporal transformer layers, each equipped with four attention heads. The hidden layer dimension is set to 512, and the embedding and representation dimensions are both set to 256. A dropout rate of 0.5 is used to mitigate overfitting.

[0145] Simulation results

[0146] The proposed method is compared with representative benchmark methods in the fields of skeleton-based emotion recognition and spatiotemporal modeling: 1) Wang et al. (2016) [2]: a feature engineering process using hand-designed motion descriptors. 2) Crenn et al. (2016a) [3]: an early deep learning scheme for skeleton-based emotion recognition based on convolutional neural networks. 3) Spatiotemporal Graph Convolutional Network ST-GCN (2018) [4]: ​​a graph convolutional neural network for modeling spatiotemporal joint dynamics. 4) Long Short-Term Memory Network LSTM (2019) [5]: a recurrent neural network model for capturing sequence dependencies. The experimental results are shown in Table 1. It can be seen that the method of the present invention has achieved performance superior to other methods, proving the effectiveness of the method in spatiotemporal transformer emotion recognition based on skeleton sequences.

[0147] Table 1 Performance Comparison of Representative Benchmark Methods

[0148] method accuracy Wang et al. (2016) 53.73% Crennetal. (2016a) 66.22% ST-GCN(2018) 65.62% LSTM (2019) 74.10% Method of the present invention 77.52%

[0149] References

[0150] [1]Bhattacharya, T.Mittal, R.Chandra, T.Randhavane, A.Bera, andD.Manocha, "STEP: SpatialTemporalGraphConvolutionalNetworksforEmotionPerceptionfromGaits", AAAI, vol.34, no.02, pp.1342-1350, Apr.2020.

[0151] [2]Wang,W.;Enescu,V.;andSahli,H.2016.Adaptivereal-timeemotionrecognitionfrombodymovements.TiiS5(4):18.

[0152] [3]Crenn,A.;Khan,R.A.;Meyer,A.;andBouakaz,S.2016a.Bodyexpressionrecognitionfromanimated3dskeleton.InIC3D,1–7.IEEE.

[0153] [4]SijieYan,YuanjunXiong,DahuaLin.Spatialtemporalgraphconvolutionalnetworksforskeleton-basedactionrecognition.ProceedingsoftheThirty-SecondAAAIConferenceonArtificialIntelligenceandThirtiethInnovativeApplicationsofArtificialIntelligenceConferenceandEighthAAAISymposiumonEducationalAdvancesinArtificialIntelligence.ArticleNo.:912,Pages7444–7452.

[0154] [5]Randhavane,T.;Bera,A.;Kapsaskis,K.;Bhattacharya,U.;Gray,K.;andManocha,D.2019.Identifyingemotionsfromwalkingusingaffectiveanddeepfeatures.arXiv:1906.11884.

Claims

1. An emotion recognition method based on a body motion spatiotemporal graph transformer, characterized in that, Includes the following steps: Step S1: Obtain the input video sequence V={I1,I2,...I... t ,...I T }, where T is the total number of video frames, and a pre-trained two-dimensional pose estimation model is used for each frame image I. t Perform human skeleton keypoint detection to obtain the joint point coordinate sequence P={J1,J2,...J} for each frame. t ,...J T }, where J t Let be the coordinate matrix of the K key points in the t-th frame, where each key point contains two-dimensional spatial coordinates (x, y) and detection confidence c; Step S2: Construct an undirected graph G for each frame based on the joint topology defined by human physiological structure. t =(V t E t ), where the node set V t For each of the K key points, the node features are initialized as (x, y, c), and the edge set E t Based on the definition of physical connections between joints, a corresponding adjacency matrix A is constructed to characterize the spatial adjacency relationships between joints. Step S3: The skeletal diagram sequence {G1,G2,...,G...} constructed in step S2 is then processed. T The node and its corresponding adjacency matrix A are input into a graph neural network. By stacking multiple graph convolutional layers or graph attention layers, neighborhood information is aggregated along the edges defined by the adjacency matrix. This allows the features of each node to gradually fuse with the spatial structure information of its multi-hop neighbor nodes, generating a joint feature representation Z that integrates local and global spatial structures. t ∈ N×d_model Where d_model is the feature embedding dimension; Step S4: Calculate the spatial coding features Z of each frame output from step S3. t Flattened into a vector vec(Z) t )∈ D Where D = K × d_model, and is associated with the learnable temporal location encoding vector p of the corresponding frame. t ∈ D Adding them together yields a spatiotemporal joint representation vector z that integrates spatial neighborhood aggregation features and temporal location information. t =vec(Z t )+p t This leads to the construction of a spatiotemporal token sequence X=[z1;z2;...;z T ]∈ T×D ; Step S5: Input the spatiotemporal token sequence X constructed in Step S4 into a graph structure transformer encoder consisting of L stacked layers. Each layer contains a multi-head self-attention module and a feedforward neural network module. The attention weights between any two tokens are calculated through the multi-head self-attention mechanism to dynamically construct global dependencies across frames and joints, forming an adaptive spatiotemporal graph structure and outputting high-level spatiotemporal semantic features. ; Step S6: Calculate the high-level spatiotemporal semantic features H output from step S5. L The data is input to a classification head, which is a multilayer perceptron. First, the features are globally pooled or specific frame features are extracted. Then, they are mapped to the sentiment category space through a fully connected layer. Finally, the sentiment category probability distribution is output through a softmax function to complete sentiment recognition.

2. The emotion recognition method based on a body motion spatiotemporal graph transformer according to claim 1, characterized in that, The pose estimation model mentioned in step S1 can be any one of YOLO-Pose, OpenPose, HRNet or MediaPipe. During detection, the human body with the highest confidence in the image is selected as the target object, and the coordinates and confidence information of its K joints are extracted.

3. The emotion recognition method based on a body motion spatiotemporal graph transformer according to claim 1, characterized in that, The joint topology relationships described in step S2 include the following connections: head-neck, neck-shoulder, shoulder-elbow, elbow-wrist, hip-knee, knee-ankle, and the connection between the trunk center and the root nodes of the limbs. The adjacency matrix A is a symmetric matrix, and self-loops are introduced in the graph convolution operation, that is, each node is connected to itself.

4. The emotion recognition method based on a body motion spatiotemporal graph transformer according to claim 1, characterized in that, The graph neural network mentioned in step S3 is any one or a combination of graph convolutional networks, graph attention networks, or graph transformers; when a graph convolutional network is used, the node feature update formula is: ； Where A is the adjacency matrix, H (l) Let W be the feature matrix of the nodes in the l-th layer. (l) Let σ be the learnable weight matrix, and σ be the non-linear activation function. The graph neural network described in step S3 has 2 to 4 stacked layers, and the output feature dimension d_model of each layer ranges from 128 to 1024. Through multi-layer mapping, the feature abstraction from local joints to body parts is realized step by step.

5. The emotion recognition method based on a body motion spatiotemporal graph transformer according to claim 1, characterized in that, The learnable temporal position encoding vector p_t mentioned in step S4 is a parameter vector that is randomly initialized and optimized together with the network parameters during model training, and is used to encode inter-frame temporal sequence information; or in another embodiment, a fixed position encoding of sine and cosine functions is used instead of learnable encoding.

6. The emotion recognition method based on a body motion spatiotemporal graph transformer according to claim 1, characterized in that, The calculation process for each layer of the graph structure converter encoder in step S5 includes: Sub-step S5-1: For input H (l-1) After layer normalization, the input is given to the multi-head self-attention module, and the attention output is calculated: H' (l) =MSA(LN(H (l-1) ))+H (l-1) ); Where MSA is a multi-head self-attention mechanism, LN is layer normalization, and + indicates residual connection; Step S5-2: For H' (l) After layer normalization, the input is fed into the feedforward neural network module to calculate the final output: H (l) =FFN(LN(H' (l) ))+H' (l) FFN is a feedforward network consisting of two fully connected layers and an activation function. The calculation formula for the multi-head self-attention mechanism is as follows: Where Q, K, and V are obtained from the input features through linear transformation, and d k The attention head is the dimension; the outputs of multiple heads are concatenated and then linearly transformed to obtain the final attention output. In step S5, the number of stacked layers L of the graph structure transformer encoder is 4 to 12, the number of attention heads per layer is 4 to 16, the dimension of the hidden layer is 256 to 1024, and a dropout rate of 0.1 to 0.5 is used to prevent overfitting.

7. The emotion recognition method based on a body motion spatiotemporal graph transformer according to claim 1, characterized in that, In step S6, the classification head first performs global average pooling on H^L or takes the features of specific frames in the sequence as video-level global features, and then maps them to the emotion category space through a multilayer perceptron containing 1 to 3 fully connected layers. The number of neurons in the output layer is equal to the number of emotion categories C, and the softmax function is used to convert the output into a probability distribution. The emotional category C includes at least four of the following: anger, neutrality, happiness, and sadness, or may be extended to include more basic emotional categories such as fear, surprise, and disgust.

8. The emotion recognition method based on a body motion spatiotemporal graph transformer according to claim 1, characterized in that, The method described above uses the cross-entropy loss function as the optimization objective during training. Where y c One-hot encoding for the real label, ŷ c Let C be the probability of the c-th category predicted by the model, where C is the total number of sentiment categories.

9. An emotion recognition system based on a body motion spatiotemporal graph transformer, characterized in that, include: The skeletal keypoint detection module is used to extract the coordinates and confidence scores of human joint points in each frame of the input video. The skeleton graph construction module is used to construct the graph structure of each frame and generate an adjacency matrix based on the physical connection relationship of the joints. The spatial coding module is used to aggregate neighborhood information from the skeleton map through a graph neural network to generate joint feature representations that fuse spatial structures. The spatiotemporal coding module is used to fuse spatial features with learnable temporal location codes to construct a spatiotemporal joint representation sequence; The graph structure transformer module is used to perform global dependency modeling on spatiotemporal representation sequences through a multi-head self-attention mechanism and output high-level semantic features. The classification module maps high-level semantic features to sentiment categories and outputs the recognition results. The graph structure transformer module adopts a multi-head self-attention mechanism, which does not rely on a fixed adjacency matrix, but dynamically calculates the attention weight between any two tokens based on the input features to form an adaptive spatiotemporal graph structure, thereby realizing the modeling of long-range dependencies across frames and joints. The spatial coding module employs graph convolutional networks or graph attention networks, and achieves hierarchical feature abstraction from local joints to whole-body posture by stacking multiple layers of networks.

10. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the emotion recognition method as described in any one of claims 1 to 8.

Images