Gait emotion recognition method, system, storage medium, and computer equipment based on spatiotemporal graph convolution.

Abstract

This invention relates to a gait emotion recognition method, system, storage medium, and computer device based on spatiotemporal graph convolution. The method includes the following steps: S1, data augmentation by reversing the temporal direction of gait; S2, obtaining deep emotion features and prior emotion features respectively through a spatiotemporal graph convolutional network and prior feature statistical methods; S3, performing nonlinear mapping on the prior emotion features using a feature mapping layer; S4, inputting the fused features of the deep emotion features and prior emotion features into an emotion classifier to obtain the emotion category. The feature mapping layer of this invention achieves more effective feature fusion by performing nonlinear mapping on prior features; it also introduces causal temporal convolution to replace general temporal convolution, effectively extracting fine-grained temporal features by enhancing temporal correlation and cross-period feature fusion. Furthermore, a walking direction recognition auxiliary task is designed to accelerate the training and convergence speed of the model, enhancing the ability to extract temporal-dependent features and the performance of emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and in particular to a gait emotion recognition method, system, storage medium, and computer device based on spatiotemporal graph convolution. Background Technology

[0002] Human emotion recognition is one of the most popular research areas among artificial intelligence researchers, playing a crucial role in human-computer interaction and social behavior tasks. Emotional states can be expressed through various means, including facial expressions, speech, physiological signals, body posture, and movement. As an individual's walking style, human gait reflects the functional state of various levels of the nervous system, providing important information about an individual's emotions. Compared to other patterns, gait can be acquired remotely without wearing protective gear, and is difficult to imitate or forge. Utilizing gait to detect individual states is a non-invasive and low-cost method. Previous research has explored the relationship between gait and emotion, showing that an individual's walking style can reflect their emotional state. For example, walking speed is closely related to individual emotions; anger is accompanied by a heavier stride, happiness by a faster gait, and sadness and depression by the slowest stride. Furthermore, arm swing and vertical head movement are gait characteristics of sadness and depression. In summary, human gait characteristics can reflect individual emotions and have significant research value.

[0003] In recent years, with the rise of deep learning, numerous models have been used for gait emotion perception. These models can be divided into two categories: traditional machine learning methods and deep learning methods. In traditional machine learning methods, most utilize statistical methods to obtain emotional features such as joint speed and angles, employ Fourier transforms to extract time-frequency features of gait sequences, and then input these transformed features into classifiers such as Naive Bayes and Support Vector Machines to obtain the final emotion category. However, traditional machine learning methods are not well-suited for learning latent spatial relationships from the non-Euclidean graph structure of gait. In contrast, deep learning methods allow for the explicit definition and utilization of spatial information from the gait skeleton. To encode graph structure information, some methods encode the spatiotemporal information of the gait skeleton into a regular image-level structure, using Convolutional Neural Networks (CNNs) for spatiotemporal feature extraction. With the advent of Graph Convolutional Networks (GCNs), it has become possible to directly extract graph structure information using convolution operations. Using GCNs to extract spatial features and temporal graph convolutions (TCNs) to aggregate temporal features is currently the mainstream framework for gait emotion recognition methods. However, most current methods are improvements based on action recognition frameworks, which focus more on extracting graph structure features. Gait emotion recognition belongs to the fine-grained classification field, where the state of emotion is more reflected in the temporal movement of gait. Better gait temporal feature extraction methods are beneficial for achieving higher accuracy in emotion recognition performance. Meanwhile, current mechanisms for fusing deep features with prior emotion features are not very effective and exhibit performance instability. The addition of prior emotion features is essentially a supplement to deep features; however, in practical applications, prior features can introduce certain perturbations into the network results, and the two types of features are not effectively combined.

[0004] In summary, there is still room for improvement in current gait emotion recognition methods based on spatiotemporal graph convolution. Summary of the Invention

[0005] To address the technical problems existing in current technologies, this invention provides a gait emotion recognition method based on spatiotemporal graph convolution. It proposes introducing causal temporal convolution to replace general temporal convolution, achieving effective extraction of fine-grained temporal features by enhancing temporal correlation and cross-period feature fusion. Simultaneously, based on the gait emotion consistency assumption—that forward and reverse gait can reflect the same emotional state—a walking direction recognition auxiliary task is designed to accelerate model training and convergence, enhancing temporal dependency extraction capabilities and emotion recognition performance. To address the insufficient fusion of deep features and prior emotion features, a feature mapping layer is proposed. Through nonlinear mapping, prior emotion features are first mapped to high-dimensional features. During backpropagation, a suitable representation space is found for the prior emotion features to fuse with deep features, jointly improving model performance.

[0006] The method of this invention is implemented using the following technical solution: a gait emotion recognition method based on spatiotemporal graph convolution, comprising the following steps:

[0007] S1. Data augmentation by reversing the gait timing direction;

[0008] S2. Deep sentiment features and prior sentiment features are obtained by using a spatiotemporal graph convolutional network and prior feature statistical calculation, respectively.

[0009] S3. A feature mapping layer is used to perform non-linear mapping on prior sentiment features;

[0010] S4. Input the fused features of deep sentiment features and prior sentiment features into the sentiment classifier to obtain the sentiment category.

[0011] The system of this invention adopts the following technical solution: a gait emotion recognition system based on spatiotemporal graph convolution, comprising:

[0012] The data augmentation module is used to augment data by reversing the gait timing direction;

[0013] The sentiment feature acquisition module is used to acquire deep sentiment features and prior sentiment features through spatiotemporal graph convolutional networks and prior feature statistical calculations, respectively.

[0014] Nonlinear mapping module: Performs nonlinear mapping on prior sentiment features using a feature mapping layer;

[0015] The emotion category acquisition module is used to input the fused features of deep emotion features and prior emotion features into the emotion classifier to obtain the emotion category.

[0016] The present invention also proposes a storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the gait emotion recognition method of the present invention are implemented.

[0017] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the gait emotion recognition method of the present invention.

[0018] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0019] 1. This invention introduces a causal convolutional neural network to enhance the temporal dependence of gait and the information correlation during the gait cycle, and proposes a walking direction recognition auxiliary task. Based on the sentiment consistency assumption, it prompts the model to pay more attention to the sentiment-related features in the temporal sequence.

[0020] 2. To address the instability of current methods for fusing deep features and prior features, this invention proposes a feature mapping layer, which enhances the compatibility of the two types of features.

[0021] 3. By introducing a feature mapping mechanism, this invention can greatly improve the gait emotion recognition performance of the model and greatly promote the application of current gait emotion recognition algorithms. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention;

[0023] Figure 2 This is a schematic diagram of the training model architecture of the present invention. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0025] Example

[0026] like Figure 1 As shown, this embodiment of the gait emotion recognition method based on spatiotemporal graph convolution includes the following steps:

[0027] S1. Data augmentation by reversing the gait timing direction;

[0028] S2. Deep sentiment features and prior sentiment features are obtained by using a spatiotemporal graph convolutional network and prior feature statistical calculation, respectively.

[0029] S3. A feature mapping layer is used to perform non-linear mapping on prior sentiment features;

[0030] S4. Input the fused features of deep sentiment features and prior sentiment features into the sentiment classifier to obtain the sentiment category.

[0031] Specifically, in this embodiment, step S1 involves reversing the sequence along the temporal direction of the gait to obtain a reversed gait sequence.

[0032] Specifically, in this embodiment, the process of obtaining deep emotional features in step S2 includes:

[0033] S211. Obtain the spatiotemporal information of the joints in the gait data through step S1, and obtain the corresponding adjacency matrix according to the topological structure of the gait skeleton nodes; in the adjacency matrix, set 1 for joints that are connected, and set 0 for those that are not connected.

[0034] S212. Input the data into the depth-temporal graph convolutional network model, such as... Figure 2As shown; the deep spatiotemporal graph convolutional network model includes three modules, each consisting of a graph convolutional neural network (GCN) and a causal temporal convolutional neural network (CausalTCN) to extract spatiotemporal information of gait;

[0035] S213. The output features are obtained by global average pooling.

[0036] Specifically, in this embodiment, the process of obtaining prior emotional features in step S2 includes:

[0037] S221. Based on the node information of each gait diagram structure, extract the action information and motion information of the gait, as shown in Table 1 and Table 2.

[0038]

[0039] Table 1

[0040]

[0041] Table 2

[0042] Specifically, motion information is obtained by calculating the angles, distances, and enclosed areas of the main joints relative to other joints; and motion information is obtained by calculating the velocities, accelerations (first derivative of velocity), and jerkes (second derivative of velocity) of five different joints, as well as the overall gait cycle.

[0043] S222. Take the average of the total sentiment features over time to obtain the temporal mean features.

[0044] Specifically, in this embodiment, step S3 includes the following process:

[0045] The prior sentiment features from step S2 are input into the feature mapping layer. After nonlinear transformation, the features are mapped to a high-dimensional feature representation. Nonlinear mapping can filter out irrelevant sentiment features, enabling the prior sentiment features to be better combined with deep features, thereby achieving better network performance.

[0046] Specifically, in this embodiment, step S4 includes the following process:

[0047] S41. Perform joint operations on the connection features of deep sentiment features and prior sentiment features to obtain hybrid features, and input the hybrid features into two fully connected layers for sentiment classification.

[0048] S42. Input the deep emotional features into the walking direction classifier for auxiliary learning;

[0049] S43. Train the model using backpropagation to achieve gait emotion recognition.

[0050] Specifically, in this embodiment, the total loss function of the model training process in step S43 includes the emotion recognition loss function and the walking direction recognition loss function, and the loss function is the cross-entropy function.

[0051] Based on the same inventive concept, this embodiment provides a gait emotion recognition system based on spatiotemporal graph convolution, including:

[0052] The data augmentation module is used to augment data by reversing the gait timing direction;

[0053] The sentiment feature acquisition module is used to acquire deep sentiment features and prior sentiment features through spatiotemporal graph convolutional networks and prior feature statistical calculations, respectively.

[0054] Nonlinear mapping module: Performs nonlinear mapping on prior sentiment features using a feature mapping layer;

[0055] The emotion category acquisition module is used to input the fused features of deep emotion features and prior emotion features into the emotion classifier to obtain the emotion category.

[0056] Furthermore, this invention also proposes a storage medium and a computer device. The storage medium stores a computer program, which, when executed by a processor, implements steps S1-S4 of the gait emotion recognition method of this invention. The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the gait emotion recognition method of this invention, which includes the aforementioned steps S1-S4.

[0057] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A gait emotion recognition method based on spatiotemporal graph convolution, characterized in that, Includes the following steps: S1. Data augmentation by reversing the gait timing direction; S2. Deep sentiment features and prior sentiment features are obtained by using spatiotemporal graph convolutional networks and prior sentiment feature statistical methods, respectively. S3. A feature mapping layer is used to perform non-linear mapping on prior sentiment features; S4. Input the fused features of deep sentiment features and prior sentiment features into the sentiment classifier to obtain the sentiment category; The process of obtaining deep sentiment features in step S2 includes: S211. Obtain the spatiotemporal information of the joints in the gait data through step S1, and obtain the corresponding adjacency matrix according to the topological structure of the gait skeleton nodes; in the adjacency matrix, set 1 for joints that are connected, and set 0 for those that are not connected. S212. Input the data into the deep spatiotemporal graph convolutional network model; wherein, the deep spatiotemporal graph convolutional network model includes three modules, each module consisting of a graph convolutional neural network GCN and a causal temporal convolutional neural network CausalTCN, to extract spatiotemporal information of gait; S213. The output features are obtained by global average pooling.

2. The gait emotion recognition method based on spatiotemporal graph convolution according to claim 1, characterized in that, The specific process of step S1 is as follows: reverse the sequence along the temporal direction of the gait to obtain the reversed gait sequence.

3. The gait emotion recognition method based on spatiotemporal graph convolution according to claim 1, characterized in that, The process of obtaining prior sentiment features in step S2 includes: S221. Based on the node information of each gait diagram structure, extract the action information and motion information of the gait; obtain the action information by calculating the angle, distance and area formed by the main joints relative to other joints; obtain the motion information by calculating the velocity, acceleration and jerk of five representative joints and the overall gait cycle. S222. Take the average of the total sentiment features over time to obtain the temporal mean features.

4. The gait emotion recognition method based on spatiotemporal graph convolution according to claim 1, characterized in that, The specific process of step S3 includes: inputting the prior sentiment features from step S2 into the feature mapping layer, and mapping the features to a high-dimensional feature representation through nonlinear transformation.

5. The gait emotion recognition method based on spatiotemporal graph convolution according to claim 1, characterized in that, The specific process of step S4 includes: S41. Combine the deep sentiment features and prior sentiment features to obtain the fused features, and input the fused features into two fully connected layers for sentiment classification. S42. Input the deep emotional features into the walking direction classifier for auxiliary training; S43. Train the model using backpropagation to achieve gait emotion recognition.

6. The gait emotion recognition method based on spatiotemporal graph convolution according to claim 5, characterized in that, The total loss function in the model training process in step S43 includes the emotion recognition loss function and the walking direction recognition loss function, and the loss function adopts cross-entropy.

7. A gait emotion recognition system based on the spatiotemporal graph convolution-based gait emotion recognition method of claim 1, characterized in that, include: The data augmentation module is used to augment data by reversing the gait timing direction; The sentiment feature acquisition module is used to acquire deep sentiment features and prior sentiment features through a spatiotemporal graph convolutional network and a prior feature statistical method, respectively. Nonlinear mapping module: Performs nonlinear mapping on prior sentiment features using a feature mapping layer; The emotion category acquisition module is used to input the fused features of deep emotion features and prior emotion features into the emotion classifier to obtain the emotion category.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the gait emotion recognition method according to any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the gait emotion recognition method according to any one of claims 1-6.

Images