Gait emotion recognition method and device based on transformer, electronic equipment and storage medium

Abstract

This invention relates to a Transformer-based gait emotion recognition method, apparatus, electronic device, and storage medium. The Transformer-based gait emotion recognition method of this invention includes: acquiring a gait video of an emotion to be identified; preprocessing the gait video to obtain a continuous gait sequence; inputting the gait sequence into a trained gait emotion recognition network to obtain the emotion category corresponding to the gait video; wherein the gait emotion recognition network is an autoencoder network model based on Spatial-Temporal Transformer. The Transformer-based gait emotion recognition method of this invention introduces and applies the Transformer algorithm, designing an autoencoder network model based on Spatial-Temporal Transformer, which significantly improves the algorithm performance.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a gait emotion recognition method, device, electronic device, and storage medium based on Transformer. Background Technology

[0002] Compared to other emotion recognition methods, gait has advantages such as long distance, difficulty in being deceived and hidden, and focusing on whole-body movements rather than local features, making it relatively less susceptible to occlusion.

[0003] Compared to traditional machine learning methods, deep learning-based methods significantly improve recognition performance. Randhavane et al. proposed an Emotion-Gait dataset containing four emotion labels (happy, sad, angry, or neutral) and used LSTM for emotion recognition. Venkatraman et al. proposed ProxEmo, a novel end-to-end emotion prediction algorithm applied to socially aware robot navigation, predicting pedestrian perceived emotions through gait. At AAAI-2020, Bhattacharya et al. applied a spatial temporal graph convolutional network (ST-GCN) architecture in STEP and performed data augmentation using annotated synthetic gaits generated by a variational autoencoder (CVAE). At ECCV-2020, Bhattacharya et al. proposed a semi-supervised method based on an autoencoder in TAEW, first training a GRU-based autoencoder for unsupervised learning on an unlabeled dataset, and then performing supervised learning on a labeled dataset, achieving an average accuracy of 84% on the Emotion-Gait benchmark dataset.

[0004] This paper addresses some unresolved issues in current research on deep learning methods based on small datasets. Existing emotion gait datasets are too small to meet the training sample requirements of deep learning. Training solely with these datasets yields poor results and suffers from overfitting. STEP employs data augmentation to expand the data. However, this method suffers from inconsistent data quality and introduces noise. Furthermore, theoretically, data augmentation based on existing datasets cannot introduce new knowledge, resulting in limited effectiveness. TAEW pre-trains by introducing other gait-related datasets for unsupervised learning. However, this method only focuses on low-level motion features and fails to capture high-order semantic features. For gait representation, TAEW uses the traditional GRU recurrent neural network, which focuses more on temporal information and ignores spatial connections between joints. STEP uses a temporal-space graph convolutional network (ST-GCN), but this method still has room for improvement. It uses manually defined rules to divide joint regions to obtain connections between joints, but these manually defined rules cannot accurately reflect the objective connections between joints in actual movement. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a gait emotion recognition method, device, electronic device and storage medium based on Transformer, introduce and apply the Transformer algorithm, and design an autoencoder network model based on Spatial-TemporalTransformer, which significantly improves the algorithm performance.

[0006] In a first aspect, the present invention provides a gait emotion recognition method based on Transformer, comprising the following steps:

[0007] Acquire gait videos of the emotion to be identified;

[0008] The gait video is preprocessed to obtain a continuous gait sequence;

[0009] The gait sequence is input into a trained gait emotion recognition network to obtain the emotion category corresponding to the gait video.

[0010] The gait emotion recognition network is an autoencoder network model based on Spatial-Temporal Transformer.

[0011] Furthermore, the training steps of the gait emotion recognition network include:

[0012] Pre-training phase: Using a gait recognition database without emotion labels, training labels for different training tasks are generated from the data itself to complete self-supervised multi-task learning.

[0013] Fine-tuning phase: Using a dataset with gait emotion labels, supervised learning methods are employed to train the deep network of the encoder and the classifier, ultimately completing model training.

[0014] Furthermore, the fine-tuning stage of the gait emotion recognition network includes:

[0015] Freeze a portion of the encoder network layers of the pre-trained model;

[0016] The gait sequence of Emotion-Gait and four types of gait emotion labels are input into the pre-trained gait emotion recognition network, and supervised learning is used to complete the supervised learning-based fine-tuning training.

[0017] Furthermore, the pre-training phase of the gait emotion recognition network includes:

[0018] The OUMVLP-Pose disordered gait sequence and the correct sequence order label are input into the gait emotion recognition network, and the original input gait sequence is reconstructed by learning the disordered gait sequence.

[0019] The data of OUMVLP-Pose after being transformed by different transformation methods and the corresponding transformation type labels are input into the gait emotion recognition network. Then, the encoder extracts the feature space and the classifier learns to determine what kind of transformation the sequence is.

[0020] The Kinectcs gait sequence and the action type labels provided in the database are input into the gait emotion recognition network. After being encoded by the encoder, the classifier learns to determine the action type of the gait sequence.

[0021] Furthermore, the gait emotion recognition network includes a Spatial-Temporal Transformer autoencoder, an average pooling layer, a two-dimensional convolutional layer, a fully connected layer, and a Softmax layer;

[0022] The Spatial-Temporal Transformer autoencoder consists of a Spatial Transformer and a Temporal Transformer, and is used to extract spatial information between key points in the same frame and temporal dimension information between different frames.

[0023] Further, the gait sequence is input into a trained gait emotion recognition network to obtain the emotion category corresponding to the gait video, including:

[0024] The gait sequence is sequentially input into the Spatial-Temporal Transformer autoencoder, the average pooling layer, and the two-dimensional convolutional layer for feature space extraction to obtain the feature space vector corresponding to the gait sequence.

[0025] The feature space vector is input into a fully connected layer and passed through a Softmax activation function to obtain the probability of the predicted sentiment type corresponding to the gait sequence.

[0026] The emotion type with the highest output probability is the emotion recognition result corresponding to the gait video.

[0027] Furthermore, the gait sequence includes multiple gait sequence frames, each containing 3D pose information for 18 joints.

[0028] Secondly, the present invention also provides a gait emotion recognition device based on Transformer, comprising:

[0029] The gait video acquisition module is used to acquire gait videos of the emotion to be identified.

[0030] A gait sequence acquisition module is used to preprocess the gait video to obtain a continuous gait sequence;

[0031] The emotion recognition module is used to input the gait sequence into a trained gait emotion recognition network to obtain the emotion category corresponding to the gait video;

[0032] The gait emotion recognition network is an autoencoder network model based on Spatial-Temporal Transformer.

[0033] Thirdly, the present invention also provides an electronic device, comprising:

[0034] At least one memory and at least one processor;

[0035] The memory is used to store one or more programs;

[0036] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a Transformer-based gait emotion recognition method as described in any of the first aspects of the present invention.

[0037] Fourthly, the present invention also provides a computer-readable storage medium.

[0038] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a Transformer-based gait emotion recognition method as described in any of the first aspects of the present invention.

[0039] This invention provides a Transformer-based gait emotion recognition method, device, electronic device, and storage medium. It introduces and applies the mainstream Transformer algorithm, designing an autoencoder network model based on Spatial-TemporalTransformer, significantly improving algorithm performance. It addresses the problem of insufficient datasets for gait emotion recognition based on deep learning methods by introducing other gait-related datasets and applying a self-supervised multi-task pre-training and fine-tuning learning strategy. Based on this new learning strategy, other gait-related datasets are first introduced for pre-training, thereby introducing new prior knowledge. The self-supervised multi-task pre-training method enables the network to extract both low-level detailed features and high-order semantic features, improving robustness.

[0040] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0041] Figure 1 A schematic diagram illustrating the steps of a gait emotion recognition method based on Transformer provided by the present invention;

[0042] Figure 2 This is a schematic diagram of an exemplary gait sequence;

[0043] Figure 3 This is a schematic diagram of the gait emotion recognition network structure used in one embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the self-attention mechanism of the gait emotion recognition network used in one embodiment of the present invention;

[0045] Figure 5 This is a flowchart illustrating the learning strategy of the present invention in one embodiment;

[0046] Figure 6 This is a roadmap of the technical solution for pre-training a gait emotion recognition network in one embodiment of the present invention;

[0047] Figure 7 This is a roadmap of the technical solution for fine-tuning and training a gait emotion recognition network in one embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of the structure of a gait emotion recognition device based on Transformer provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0050] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0051] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0052] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0053] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0054] To address the problems in the background art, embodiments of this application provide a gait emotion recognition method based on Transformer, such as... Figure 1 As shown, the method includes the following steps:

[0055] S01: Acquire gait video of the emotion to be identified.

[0056] Compared with other methods (face, physiological signals, etc.), gait emotion recognition has advantages such as long distance, resistance to occlusion and deception, simple and convenient data acquisition equipment (camera), and wide range of application scenarios, and has broad application prospects.

[0057] S02: Preprocess the gait video to obtain a continuous gait sequence.

[0058] In a preferred embodiment, a state-of-the-art pose estimation algorithm is used to extract pose sequences from an RGB image, for example, a gait sequence at a 45-degree angle, such as... Figure 2 As shown.

[0059] The gait sequence consists of multiple gait sequence frames, each containing 3D pose information for 18 joints.

[0060] S03: Input the gait sequence into the trained gait emotion recognition network to obtain the emotion category corresponding to the gait video. The gait emotion recognition network is an autoencoder network model based on Spatial-Temporal Transformer.

[0061] In a preferred embodiment, such as Figure 3 As shown, the gait emotion recognition network includes a Spatial-Temporal Transformer autoencoder, an average pooling layer, a two-dimensional convolutional layer, a fully connected layer, and a Softmax layer.

[0062] The Spatial-Temporal Transformer autoencoder consists of a Spatial Transformer and a Temporal Transformer, and is used to extract spatial information between key points in the same frame and temporal dimension information between different frames.

[0063] Currently, the most widely used method for action recognition based on 3D pose skeletal maps is Graph Neural Networks (GNNs), especially Graph Convolutional Networks (GCNs), because they can effectively capture spatial (within a single frame of a sequence) and temporal (between frames) information. Therefore, this patent will also adopt a method based on graph convolutional networks. Previous studies, such as those based on ST-GCNs, have fixed topological mappings of joints, which may hinder the mining of rich potential information in spatiotemporal gait sequences. Furthermore, the joint region division based on manually set rules to obtain the connections between joints cannot truly reflect the objective connections between joints in actual movement. Secondly, implementing spatial and temporal convolutions starting from standard 2D convolutions is, to some extent, limited by the size of the convolution kernel. For example, the correlation between actions such as "clapping" and the correlation between unlinked body joints in the human skeleton (e.g., left and right hands) are also underestimated. Therefore, we address these limitations by using Spatial-Temporal Transformer. By employing a Transformer-based encoder, we can better mine the latent information in the gait sequence, extracting temporal information between different frames and spatial information between joints. This makes the model flexible, allowing the relative importance of joints to adapt to each input gait sequence. For example, the connection strength between joints is determined by the Transformer's self-attention mechanism, rather than by pre-defining the adjacency matrix as in the typical formulas previously used in GCN-based models.

[0064] like Figure 3 As shown, the Spatial Transformer module is designed to extract high-dimensional features of the embedding from a single gait sequence frame. The input data consists of 3D pose information for 18 joints. Each joint (i.e., 3 coordinate values) is treated as a patch, and feature extraction is performed across all patches following the general visual Transformer process. First, we use a trainable Linear Projection to map the coordinates of each joint to a high-dimensional space; this is called Spatial Patch Embedding. We then add the Patch Embedding to a learnable Spatial Positional Embedding. The generated joint sequence features are input into the Spatial Transformer Encoder. As shown in the figure, this structure consists of a self-attention layer and a layer normalization layer. This encoder applies a self-attention mechanism to integrate information from all joints, thereby automatically learning the spatial information between joints. The self-attention mechanism is as follows... Figure 4As shown. A self-attention mechanism is used on the skeleton graph, including: (1) For each body joint, calculate query Q, key K and value V. (2) Perform a dot product of Q and K for each to obtain V, which represents the connection strength between each pair of nodes. (3) Finally, each node is scaled according to its relevance. (4) The weighted nodes are summed together to obtain their new features.

[0065] Based on the gait emotion recognition network described above, identifying the corresponding emotion category from the gait sequence includes the following sub-steps:

[0066] S031: The gait sequence is sequentially input into the Spatial-Temporal Transformer autoencoder, the average pooling layer, and the two-dimensional convolutional layer to extract the feature space and obtain the feature space vector corresponding to the gait sequence.

[0067] S032: Input the feature space vector into a fully connected layer and pass it through the Softmax activation function to obtain the probability of the predicted sentiment type corresponding to the gait sequence.

[0068] S033: Output the emotion type with the highest probability as the emotion recognition result corresponding to the gait video.

[0069] In a preferred embodiment, the learning strategy flowchart is as follows: Figure 5 As shown, the training steps of the gait emotion recognition network include the following sub-steps:

[0070] S11: Pre-training phase: Using a gait recognition database without emotion labels, training labels for different training tasks are generated from the data itself to complete self-supervised multi-task learning.

[0071] S12: Fine-tuning stage: Using a dataset with gait emotion labels, supervised learning is used to train the deep network of the encoder and the classifier, and finally the model training is completed.

[0072] like Figure 6 As shown, in the pre-training phase, a large-scale gait recognition database (OUMVLP-Pose) or action recognition database (Kinectcs) without emotion labels is used. Based on different training tasks, training labels for the corresponding tasks are generated from the data itself, completing self-supervised multi-task learning. This allows the network to extract both low-level detailed features and high-order semantic features while improving robustness. In the fine-tuning phase, a dataset with gait emotion labels is used, employing supervised learning methods to train the deep network of the encoder and the classifier, ultimately completing model training.

[0073] The OUMVLP-Pose multi-view large-group pose sequence database contains 10,307 back-and-forth walking sequences captured by seven webcams at 15° intervals. Kinectcs is a large-scale, high-quality motion dataset containing up to 650,000 video clips covering 400 to 700 human action classes. These videos include human-object interactions such as playing musical instruments, and human-to-human interactions such as shaking hands and hugging. Each action class has at least 400 to 700 video clips. Each clip is manually annotated by an action class and lasts approximately 10 seconds.

[0074] In real life, both single-label and multi-label images often contain not only simple visual information but also various semantic information. Traditional autoencoders aim to reconstruct the input, while self-supervised learning learns distinguishable visual features by designing auxiliary tasks. The labels used in self-supervised learning training can be directly obtained from the training data, similar to supervised learning in that labels provide supervision for the algorithm model's training. However, self-supervised learning completes its learning tasks by mining the properties of the data to generate semantic labels for visual features, which is the difference from supervised learning. This patent employs a self-supervised learning method to learn from a gait-related database without emotion labels. By designing different self-supervised learning tasks, the model can mine more potential gait features, improving the model's recognition performance while addressing the overfitting problem of small-scale datasets in deep learning.

[0075] Specifically, the pre-training phase of self-supervised learning includes the following sub-steps:

[0076] S111: Input the scrambled gait sequence of OUMVLP-Pose and the correct sequence order label into the gait emotion recognition network, and reconstruct the original input gait sequence by learning the scrambled gait sequence.

[0077] S112: Input the data of OUMVLP-Pose after being transformed by different transformation methods and the corresponding transformation type labels into the gait emotion recognition network, and then extract the feature space through the encoder, and the classifier learns to determine what kind of transformation type the sequence is.

[0078] S113: Input the Kinectcs gait sequence and the action type label provided in the database into the gait emotion recognition network. After encoding by the encoder, the classifier learns to determine the action type of the gait sequence.

[0079] like Figure 6As shown, in Task 1, gait sequence reconstruction, the input data is an OUMVLP-Pose scrambled (e.g., reversed or shuffled) gait sequence and the correct sequence order labels. By learning to reconstruct the original input gait sequence from the scrambled gait sequence, the model can learn the inherent temporal correlations embedded in the gait sequence during gait encoding, while also learning low-level detailed features. In Task 2, transformation type determination, the input data is OUMVLP-Pose data transformed by different transformation methods (e.g., rotation, reshuffling, scaling, etc.) and the corresponding transformation type labels. Then, the encoder extracts the feature space, and the classifier learns to determine the transformation type of the sequence, thereby improving the model's robustness. In Task 3, action recognition, the input data is a Kinectcs gait sequence and the action type labels provided in the database. After encoding by the encoder, the classifier learns to determine the action type of the gait sequence (e.g., walking, running, long jump, etc.), enabling the model to mine higher-order gait semantic features.

[0080] The fine-tuning phase of supervised learning used the Emotion-Gait dataset. Emotion-Gait contains 2177 gait sequences, including 342 actually collected emotional gait sequences. Ninety participants walked while contemplating four different emotions (anger, neutrality, happiness, and sadness), with each participant walking a total distance of 7 meters. The remaining 1835 gait sequences came from the Edinburgh Locomotion MOCAP Database and were labeled with the four emotion tags by the same domain expert. These videos were also labeled by the domain expert.

[0081] Specifically, such as Figure 7 As shown, the fine-tuning stage includes the following sub-steps:

[0082] S121: Freeze a portion of the encoder network layers of the pre-trained model.

[0083] S122: Input the gait sequence of Emotion-Gait and the four types of gait emotion labels into the pre-trained gait emotion recognition network, and complete the supervised learning-based fine-tuning training.

[0084] This application also provides a gait emotion recognition device based on Transformer, such as... Figure 8 As shown, the Transformer-based gait emotion recognition device 400 includes:

[0085] Gait video acquisition module 401 is used to acquire gait videos of the emotion to be identified;

[0086] The gait sequence acquisition module 402 is used to preprocess the gait video to obtain a continuous gait sequence;

[0087] The emotion recognition module 403 is used to input the gait sequence into a trained gait emotion recognition network to obtain the emotion category corresponding to the gait video; wherein, the gait emotion recognition network is an autoencoder network model based on Spatial-TemporalTransformer.

[0088] Preferably, the training steps of the gait emotion recognition network include:

[0089] Pre-training phase: Using a gait recognition database without emotion labels, training labels for different training tasks are generated from the data itself to complete self-supervised multi-task learning.

[0090] Fine-tuning phase: Using a dataset with gait emotion labels, supervised learning methods are employed to train the deep network of the encoder and the classifier, ultimately completing model training.

[0091] Preferably, the fine-tuning stage of the gait emotion recognition network includes:

[0092] Freeze a portion of the encoder network layers of the pre-trained model;

[0093] The gait sequence of Emotion-Gait and four types of gait emotion labels are input into the pre-trained gait emotion recognition network, and supervised learning is used to complete the supervised learning-based fine-tuning training.

[0094] Preferably, the pre-training phase of the gait emotion recognition network includes:

[0095] The OUMVLP-Pose disordered gait sequence and the correct sequence order label are input into the gait emotion recognition network, and the original input gait sequence is reconstructed by learning the disordered gait sequence.

[0096] The data of OUMVLP-Pose after being transformed by different transformation methods and the corresponding transformation type labels are input into the gait emotion recognition network. Then, the encoder extracts the feature space and the classifier learns to determine what kind of transformation the sequence is.

[0097] The Kinectcs gait sequence and the action type labels provided in the database are input into the gait emotion recognition network. After being encoded by the encoder, the classifier learns to determine the action type of the gait sequence.

[0098] Preferably, the gait emotion recognition network includes a Spatial-Temporal Transformer autoencoder, an average pooling layer, a two-dimensional convolutional layer, a fully connected layer, and a Softmax layer;

[0099] The Spatial-Temporal Transformer autoencoder consists of a Spatial Transformer and a Temporal Transformer, and is used to extract spatial information between key points in the same frame and temporal dimension information between different frames.

[0100] Preferably, the emotion recognition module includes:

[0101] The feature space vector extraction unit is used to sequentially input the gait sequence into the Spatial-TemporalTransformer autoencoder, the average pooling layer, and the two-dimensional convolutional layer to extract the feature space and obtain the feature space vector corresponding to the gait sequence.

[0102] The probability prediction unit is used to input the feature space vector into the fully connected layer and pass it through the Softmax activation function to obtain the probability of the predicted sentiment type corresponding to the gait sequence.

[0103] The result output unit is used to output the emotion recognition result corresponding to the gait video with the highest probability emotion type.

[0104] Preferably, the gait sequence includes multiple gait sequence frames, and each individual gait sequence frame contains 3D pose information for 18 joints.

[0105] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0106] This application also provides an electronic device, including:

[0107] At least one memory and at least one processor;

[0108] The memory is used to store one or more programs;

[0109] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the Transformer-based gait emotion recognition method as described above.

[0110] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0111] This application also provides a computer-readable storage medium.

[0112] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a Transformer-based gait emotion recognition method as described above.

[0113] Computer storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0114] This invention provides a Transformer-based gait emotion recognition method, device, electronic device, and storage medium. It introduces and applies the mainstream Transformer algorithm, designing an autoencoder network model based on Spatial-TemporalTransformer, significantly improving algorithm performance. It addresses the problem of insufficient datasets for gait emotion recognition based on deep learning methods by introducing other gait-related datasets and applying a self-supervised multi-task pre-training and fine-tuning learning strategy. Based on this new learning strategy, other gait-related datasets are first introduced for pre-training, thereby introducing new prior knowledge. The self-supervised multi-task pre-training method enables the network to extract both low-level detailed features and high-order semantic features, improving robustness.

[0115] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A gait emotion recognition method based on Transformer, characterized in that, Includes the following steps: Acquire gait videos of the emotion to be identified; The gait video is preprocessed to obtain a continuous gait sequence; The gait sequence is input into a trained gait emotion recognition network to obtain the emotion category corresponding to the gait video. The gait emotion recognition network is an autoencoder network model based on Spatial-Temporal Transformer, including a Spatial-Temporal Transformer autoencoder, an average pooling layer, a two-dimensional convolutional layer, a fully connected layer, and a Softmax layer. The Spatial-Temporal Transformer autoencoder consists of a Spatial Transformer and a Temporal Transformer, used to extract spatial information between keypoints in the same frame and temporal information between different frames. The training steps of the gait emotion recognition network include: Pre-training phase: Using a gait recognition database without emotion labels, training labels for different training tasks are generated from the data itself to complete self-supervised multi-task learning. Specifically, this includes: inputting OUMVLP-Pose scrambled gait sequences and correct sequence order labels into the gait emotion recognition network, and reconstructing the original input gait sequences by learning from the scrambled gait sequences; inputting OUMVLP-Pose data transformed by different transformation methods and corresponding transformation type labels into the gait emotion recognition network, and then extracting the feature space through the encoder, with the classifier learning to determine the transformation type of the sequence; inputting Kinectcs gait sequences and the action type labels provided by the database into the gait emotion recognition network, and then encoding them through the encoder, with the classifier learning to determine the action type of the gait sequence. Fine-tuning phase: Using a dataset with gait emotion labels, supervised learning is used to train the deep network of the encoder and the classifier to complete the model training. Specifically, this includes: freezing some network layers of the encoder in the pre-trained model; inputting the gait sequence of Emotion-Gait and four types of gait emotion labels into the pre-trained gait emotion recognition network, and using supervised learning to complete the supervised learning-based fine-tuning training.

2. The gait emotion recognition method based on Transformer according to claim 1, characterized in that, The gait sequence is input into a trained gait emotion recognition network to obtain the emotion category corresponding to the gait video, including: The gait sequence is sequentially input into the Spatial-Temporal Transformer autoencoder, the average pooling layer, and the two-dimensional convolutional layer for feature space extraction to obtain the feature space vector corresponding to the gait sequence. The feature space vector is input into a fully connected layer and passed through a Softmax activation function to obtain the probability of the predicted sentiment type corresponding to the gait sequence. The emotion type with the highest output probability is the emotion recognition result corresponding to the gait video.

3. The gait emotion recognition method based on Transformer according to claim 1, characterized in that: The gait sequence includes multiple gait sequence frames, and each individual gait sequence frame contains 3D pose information for 18 joints.

4. A gait emotion recognition device based on Transformer, characterized in that, include: The gait video acquisition module is used to acquire gait videos of the emotion to be identified. A gait sequence acquisition module is used to preprocess the gait video to obtain a continuous gait sequence; An emotion recognition module is used to input the gait sequence into a trained gait emotion recognition network to obtain the emotion category corresponding to the gait video. The gait emotion recognition network is an autoencoder network model based on Spatial-Temporal Transformer, including a Spatial-Temporal Transformer autoencoder, an average pooling layer, a two-dimensional convolutional layer, a fully connected layer, and a Softmax layer. The Spatial-Temporal Transformer autoencoder consists of a Spatial Transformer and a Temporal Transformer, used to extract spatial information between keypoints in the same frame and temporal information between different frames. The training steps of the gait emotion recognition network include: Pre-training phase: Using a gait recognition database without emotion labels, training labels for different training tasks are generated from the data itself to complete self-supervised multi-task learning. This includes: inputting OUMVLP-Pose scrambled gait sequences and correct sequence order labels into the gait emotion recognition network, and reconstructing the original input gait sequences by learning from the scrambled gait sequences; inputting OUMVLP-Pose data transformed by different transformation methods and corresponding transformation type labels into the gait emotion recognition network, and then extracting the feature space through the encoder, with the classifier learning to determine the transformation type of the sequence; inputting Kinectcs gait sequences and the action type labels provided by the database into the gait emotion recognition network, and then encoding them through the encoder, with the classifier learning to determine the action type of the gait sequence. Fine-tuning phase: Using a dataset with gait emotion labels, supervised learning is used to train the deep network of the encoder and the classifier to complete the model training. This includes: freezing some network layers of the encoder of the pre-trained model; inputting the gait sequence of Emotion-Gait and four types of gait emotion labels into the pre-trained gait emotion recognition network, and using supervised learning to complete the supervised learning-based fine-tuning training.

5. An electronic device, characterized in that, include: At least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a Transformer-based gait emotion recognition method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a Transformer-based gait emotion recognition method as described in any one of claims 1-3.

Images