An emotion recognition method based on 3D skeleton gait analysis

Abstract

The application provides an emotion recognition method based on 3D skeleton gait analysis, a fusion multi-flow structure formed by a local frame flow of joint information, a global sequence flow of joint information, a local frame flow of skeleton information and a global sequence flow of skeleton information, deep features in the fusion multi-flow structure data are captured through an MSMTA-GCN network, and the fusion multi-flow structure data are complementary to each other, more information for MSMTA-GCN network recognition can be provided, and therefore the accuracy of emotion classification and recognition is improved. In the MSMTA-GCN network, an MTA-GCN block (multi-thread self-attention layer) is designed to relieve the receptive field imbalance problem. Meanwhile, the self-attention mechanism is introduced to enable the GCN to focus on the part containing more effective information in the input sequence, so as to capture the non-local dependence relationship between different joints of the human body. The method is universal and robust, and can work with the existing GCN framework to achieve superior performance in a simple way.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically involving an emotion recognition method based on 3D skeletal gait analysis. Background Technology

[0002] In recent years, emotion recognition methods based on 3D skeleton data have attracted much attention, and many high-performance methods have been proposed. 3D skeleton-based data representation methods have many advantages in recognition tasks. For example, researchers can easily extract 3D skeleton data from video data using existing algorithms; furthermore, because 3D skeleton data is independent of race, culture, or ethnic background, it has significant advantages in cross-cultural research and other related applications.

[0003] Deep learning techniques are commonly used for emotion recognition, with graph convolutional networks (GCNNs) playing a significant role in recent years. For example, GCNNs can model 3D skeletal data using the connections between joints, thereby improving the emotion recognition capabilities of 3D skeletal data-based methods. In the STEP graph convolutional model proposed in 2017, researchers used the physical structure of the human skeleton as an adjacency matrix, effectively improving the accuracy of emotion recognition. However, the difficulty in extracting effective features from joints and insufficient utilization of complementary information may limit the model's ability to capture complex emotional states and potentially lead to poor performance on certain tasks. Summary of the Invention

[0004] To address the problems existing in the current technical solutions, this invention proposes an emotion recognition method based on 3D skeletal gait analysis.

[0005] The specific technical solution adopted in this invention is as follows:

[0006] An emotion recognition method based on 3D skeletal gait analysis, characterized by the following steps:

[0007] S1. The joint motion sequence of human gait in walking video is used as raw data. After data preprocessing, the raw data is used to obtain local frame stream of joint information, global sequence stream of joint information, local frame stream of skeleton information, and global sequence stream of skeleton information, forming a fused multi-stream structure.

[0008] S2. Obtain a pre-trained MSMTA-GCN network, wherein the MSMTA-GCN network structure is composed of an MTA-GCN block, a ReLU activation function, a residual ST-GCN block, an ST-GCN block, an average pooling layer, and a convolutional layer cascaded in sequence;

[0009] S3. The local frame stream of joint information, the global sequence stream of joint information, the local frame stream of skeleton information, and the global sequence stream of skeleton information are respectively used as inputs to the MSMTA-GCN network to obtain the probability distribution of emotion prediction category under each input condition. The probability values ​​belonging to the same emotion prediction category are added together to form a new probability distribution. The new probability distribution is normalized by softmax, and the emotion prediction category with the highest probability value is taken as the emotion category finally predicted by the network to realize the recognition of human emotions.

[0010] As a preferred method, the data preprocessing method in step S1 is as follows:

[0011] S11. Establish a camera coordinate system centered on the camera and a human coordinate system centered on the human body. In the camera coordinate system, the origin is the optical center of the camera, the X-axis is parallel to the long side of the camera screen, the Y-axis is parallel to the short side of the camera screen, and the Z-axis is the optical axis of the camera. In the human coordinate system, the origin is the center of gravity of the human body, the X-axis is the shoulder, and the Y-axis is the spine.

[0012] S12. Acquire a human walking video, and continuously extract H frames of photos from the video as the joint action sequence J of the human gait. When the number of video frames is less than H frames, the entire video is extended to H frames by looping and copying it head to tail. In each frame, the body faces the same direction. All joints in the joint action sequence J are referenced to the same camera coordinate system and the same human coordinate system.

[0013] S13. Local frame stream J from joint motion sequence J to obtain joint information F J F The i-th time frame in the t-th time frame J Joint The function form is:

[0014]

[0015] Where root represents the center of gravity of the human body. It is the i-th time frame of the t-th time frame of the joint motion sequence J. J One joint, R is the center joint of the t-th time frame. t R is the rotation matrix for the t-th time frame. t The function form is:

[0016]

[0017]

[0018]

[0019] Where, α t It is the angle between the X-axis of the camera coordinate system and the X-axis of the human body coordinate system, γ. t It is the angle between the Z-axis of the camera coordinate system and the y-axis of the human coordinate system. It is based on α t The counterclockwise rotation matrix of the X-axis in the radian rotation camera coordinate system. It is based on γ t The counterclockwise rotation matrix of the Z-axis in the radian rotation camera coordinate system;

[0020] S14. Obtain the global sequence stream J of joint information from the joint motion sequence J. S J S The i-th time frame in the t-th time frame J Joint The function form is

[0021]

[0022] in, It is the center joint of the 0th time frame;

[0023] S15. Local frame stream J based on joint information F Local frame stream B that obtains skeletal information F B F The i-th time frame in the t-th time frame B A skeleton The function form is:

[0024]

[0025] in, It is the time frame t that is related to the i-th time frame. J The joints adjacent to each other;

[0026] S16. From the global sequence stream J of joint information S Obtain the global sequence stream B of skeletal information S B S The i-th time frame in the t-th time frame B A skeleton The function form is:

[0027]

[0028] Preferably, in step S2, the input data of the MTA-GCN block first passes through an attention module and a batch normalization layer, outputting feature vector E1. The input data then passes through a residual layer, outputting feature vector E2. E1 and E2 are added to obtain feature vector E3. E3 is then passed sequentially through a ReLU activation function, a batch normalization layer, another ReLU activation function, a first convolutional layer, another batch normalization layer, and a Dropout layer, outputting feature vector D1. The residual layer consists of a cascaded convolutional layer with a 1×1 kernel and a stride of (1,1) and a batch normalization layer. The first convolutional layer has a 75×1 kernel, a stride of (1,1), and padding of (37,0). The attention module includes parallel N... thread There are N threads {N1, ..., N} thread}, input data f in The input is fed into the attention module to obtain the output f. atten The function form is:

[0029]

[0030] Where, N kernel The kernel size represents the spatial dimension; w i,k N represents i The k-th weight matrix in the matrix; A k It is the adjacency matrix of the original normalized human physical structure, and A k The element values ​​in each N i N remains unchanged i ∈{N1,…,N thread};B i,k N represents i The k-th adjacency matrix in the middle; during the training process, B i,k The element values ​​are parameterized and optimized; C i,k C is a data correlation graph uniquely learned and determined for each sample. i,k The function form is:

[0031]

[0032] Where G represents a graph, θ i,k and This represents two different embedding functions, both of which are 1×1 convolutional layers, but with different strides and padding methods. The residual layer function is as follows:

[0033] f g =BN(f atten )+BN(W in f in )

[0034] Among them, W in f in The weight matrix; BN represents the batch normalization layer.

[0035] Preferably, in the MSMTA-GCN network, the ST-GCN block is composed of a GCN block, a TCN block, and a ReLU activation function cascaded sequentially; the GCN block is composed of a second convolutional layer and a graph convolutional layer cascaded sequentially; the second convolutional layer has a kernel size of 1×1, a stride of (1,1), and padding of (0,0); the TCN block is composed of a batch normalization layer, a ReLU activation function, a third convolutional layer, a batch normalization layer, and a Dropout layer cascaded sequentially; the third convolutional layer has a kernel size of 75×1, a stride of (1,1), and padding of (37,0); the input data D2 of the residual ST-GCN block first passes through a GCN block and a TCN block sequentially to output a feature vector E4; the feature vector E4 passes through a ReLU activation function to output a feature vector E5; the feature vector E4 then passes through the residual layer to output a feature vector E6; the feature vector E5 and the feature vector E6 are added to output a feature vector D3.

[0036] Preferably, the training samples of the MSMTA-GCN network are in the form of (J F J S B F B S Data for training samples, where Y represents the true class label of the training samples;

[0037] In each iteration of the MSMTA-GCN network, the input training sample data of size B×C×T×V first passes through an MTA-GCN block to obtain a feature vector D1 of size B×C1×T×V; feature vector D1 passes through a ReLU activation function to obtain a feature vector D2 of size B×C1×T×V; feature vector D2 passes through a residual ST-GCN block to obtain a feature vector of size B×C1×T×V. The feature vector D3; the feature vector D3 is processed through an ST-GCN block to obtain a size of The feature vector D4 is processed by an average pooling layer to obtain a feature vector D5 of size B×C2×1×1. Finally, the feature vector D5 is processed by a convolutional layer with a kernel size of 1×1 and a stride of (1,1) to obtain a probability distribution P of the predicted emotion category of size B×4×1×1. The probability values ​​of belonging to the same emotion category are summed and normalized by softmax. The emotion category with the highest probability value is taken as the final predicted emotion category of the network. The classification loss in the form of cross-entropy is calculated based on the true category label Y of the training samples and the probability distribution P of the predicted emotion category to update the network parameters of the MSMTA-GCN network.

[0038] Preferably, the classification loss has the following functional form:

[0039]

[0040] Where N is the number of categories of human emotions, p n and y n These are the nth-dimensional elements of P and Y, respectively.

[0041] Preferably, the training set of the MSMTA-GCN network is an emotion gait dataset, which contains 2177 joint movement sequences of real human gait, and the human emotion categories are happiness, sadness, anger or indifference.

[0042] Preferably, the MSMTA-GCN network uses five-fold cross-validation, and the ratio of training sample data to test sample data is 4:1.

[0043] As a preferred approach, the training samples of the MSMTA-GCN network are processed in batches in each iteration, with each batch containing 8 training sample data.

[0044] Preferably, the MSMTA-GCN network is set to iterate 200 times, and the initial learning rate of the MSMTA-GCN network is set to 0.01. Starting from the 100th iteration, the initial learning rate is divided by 10 every 50 iterations until 200 iterations are completed.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] (1) This invention proposes an emotion recognition method based on 3D skeletal gait analysis. The fusion multi-stream structure, which consists of local frame streams of joint information, global sequence streams of joint information, local frame streams of skeletal information, and global sequence streams of skeletal information, captures deep features in the fusion multi-stream structure data through the MSMTA-GCN network. Furthermore, the fusion multi-stream structure data complement each other and can provide more information for MSMTA-GCN network recognition, thereby improving the accuracy of emotion classification and recognition.

[0047] (2) In the MSMTA-GCN network, this invention designs an MTA-GCN block (multi-threaded self-attention layer) to alleviate the receptive field imbalance problem. At the same time, the introduction of a self-attention mechanism enables GCN to focus on the part of the input sequence that contains more effective information, thereby capturing the non-local dependencies between different joints in the human body.

[0048] (3) The method of the present invention is universal and robust and can work with existing GCN frameworks to achieve superior performance in a simple way. Attached Figure Description

[0049] Figure 1 This is a fusion of multi-stream input maps for an emotion recognition method based on 3D skeletal gait analysis;

[0050] Figure 2 This is a block structure diagram of MTA-GCN, a method for emotion recognition based on 3D skeletal gait analysis.

[0051] Figure 3 This is a schematic diagram of a single-stream input MSMTA-GCN network for an emotion recognition method based on 3D skeletal gait analysis. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0053] This invention proposes an emotion recognition method based on 3D skeletal gait analysis. The specific technical solutions adopted in this invention are as follows:

[0054] S1. The joint motion sequence of human gait in walking video is used as raw data. After data preprocessing, the raw data is used to obtain local frame stream of joint information, global sequence stream of joint information, local frame stream of skeleton information, and global sequence stream of skeleton information, forming a fused multi-stream structure.

[0055] In this embodiment of the invention, the data preprocessing method in step S1 is as follows:

[0056] S11. Establish a camera coordinate system centered on the camera and a human coordinate system centered on the human body. In the camera coordinate system, the origin is the optical center of the camera, the X-axis is parallel to the long side of the camera screen, the Y-axis is parallel to the short side of the camera screen, and the Z-axis is the optical axis of the camera. In the human coordinate system, the origin is the center of gravity of the human body, the X-axis is the shoulder, and the Y-axis is the spine.

[0057] S12. Acquire a human walking video, and continuously extract H frames of photos from the video as the joint action sequence J of the human gait. When the number of video frames is less than H frames, the entire video is extended to H frames by looping and copying it head to tail. In each frame, the body faces the same direction. All joints in the joint action sequence J are referenced to the same camera coordinate system and the same human coordinate system.

[0058] S13. Local frame stream J from joint motion sequence J to obtain joint information F J F The i-th time frame in the t-th time frame J Joint The function form is:

[0059]

[0060] Where root represents the center of gravity of the human body. It is the i-th time frame of the t-th time frame of the joint motion sequence J. J One joint, R is the center joint of the t-th time frame. t R is the rotation matrix for the t-th time frame. t The function form is:

[0061]

[0062]

[0063]

[0064] Where, α t It is the angle between the X-axis of the camera coordinate system and the X-axis of the human body coordinate system, γ. t It is the angle between the Z-axis of the camera coordinate system and the y-axis of the human coordinate system. It is based on α t The counterclockwise rotation matrix of the X-axis in the radian rotation camera coordinate system. It is based on γ t The counterclockwise rotation matrix of the Z-axis in the radian rotation camera coordinate system;

[0065] S14. Obtain the global sequence stream J of joint information from the joint motion sequence J. S J S The i-th time frame in the t-th time frame J Joint The function form is

[0066]

[0067] in, It is the center joint of the 0th time frame;

[0068] S15. Local frame stream J based on joint information F Local frame stream B that obtains skeletal information F B F The i-th time frame in the t-th time frame B A skeleton The function form is:

[0069]

[0070] in, It is the time frame t that is related to the i-th time frame. J The joints adjacent to each other;

[0071] S16. From the global sequence stream J of joint information S Obtain the global sequence stream B of skeletal information S B S The i-th time frame in the t-th time frame B A skeleton The function form is:

[0072]

[0073] It should be noted that, since the flow structure of human skeletal information can be extracted using local frame-level joints or sequence-level joints of the human body, this invention represents the frame-level joints of skeletal information as local frame streams of skeletal information, and the sequence-level joints of skeletal information as global sequence streams of skeletal information.

[0074] It should be noted that using local frame streams containing joint information as input data can help the network reduce orientation bias and make the network pay more attention to changes in local body posture and movements between different frames; introducing skeletal information can help the network extract dynamic features; the data preprocessing method in step S1 enables the MSMTA-GCN network to capture global information.

[0075] S2. Obtain a pre-trained MSMTA-GCN network, wherein the MSMTA-GCN network structure is composed of an MTA-GCN block, a ReLU activation function, a residual ST-GCN block, an ST-GCN block, an average pooling layer, and a convolutional layer cascaded in sequence;

[0076] S3. The local frame stream of joint information, the global sequence stream of joint information, the local frame stream of skeleton information, and the global sequence stream of skeleton information are respectively used as inputs to the MSMTA-GCN network to obtain the probability distribution of emotion prediction category under each input condition. The probability values ​​belonging to the same emotion prediction category are added together to form a new probability distribution. The new probability distribution is normalized by softmax, and the emotion prediction category with the highest probability value is taken as the emotion category finally predicted by the network to realize the recognition of human emotions.

[0077] In this embodiment of the invention, in the MTA-GCN block mentioned in step S2, the input data of the MTA-GCN block first passes through an attention module and a batch normalization layer, outputting a feature vector E1. The input data of the MTA-GCN block then passes through a residual layer, outputting a feature vector E2. E1 and E2 are added to obtain a feature vector E3. E3 is then passed through a ReLU activation function, a batch normalization layer, a ReLU activation function, a first convolutional layer, a batch normalization layer, and a Dropout layer in sequence, outputting a feature vector D1. The residual layer is composed of a convolutional layer with a kernel size of 1×1 and a stride of (1,1) and a batch normalization layer cascaded together. The first convolutional layer has a kernel size of 75×1, a stride of (1,1), and padding of (37,0). The attention module includes parallel N... thread There are N threads {N1, ..., N} thread}, input data f in The input is fed into the attention module to obtain the output f. atten The function form is:

[0078]

[0079] Where, N kernel The kernel size represents the spatial dimension; w i,k N represents i The k-th weight matrix in the matrix; A k It is the adjacency matrix of the original normalized human physical structure, and A k The element values ​​in each N i N remains unchanged i ∈{N1,…,N thread};B i,k N represents i The k-th adjacency matrix in the middle; during the training process, B i,kThe element values ​​are parameterized and optimized; C i,k C is a data correlation graph uniquely learned and determined for each sample. i,k The function form is:

[0080]

[0081] Where G represents a graph, θ i,k and This represents two different embedding functions, both of which are 1×1 convolutional layers, but with different strides and padding methods. The residual layer function is as follows:

[0082] f g =BN(f atten )+BN(W in f in )

[0083] Among them, W in f in The weight matrix; BN represents the batch normalization layer.

[0084] It should be noted that, in order to determine whether a connection exists between two human joints and the strength of that connection, this embodiment of the invention uses a normalized embedded Gaussian function to calculate the similarity between the two human joints. Since the normalized embedded Gaussian function is equivalent to the softmax operation in the experiment, C... i,k It can be rewritten as:

[0085]

[0086] in, Represents the embedding function θ i,k The transpose of the weight matrix, Represents embedded functions The weight matrix, specifically, given input data f of size C×T×V. in First f in After two embedding functions θ i,k and Then, V×C is obtained. e Embedded feature map of T matrix and C e The embedding feature maps of the T×V matrix are then multiplied to obtain a data correlation map C of size V×V. i,k In this embodiment of the invention, a 1×1 convolutional layer is selected as the embedding function.

[0087] In the MSMTA-GCN network, the ST-GCN block is composed of a GCN block, a TCN block, and a ReLU activation function cascaded sequentially; the GCN block is composed of a second convolutional layer and a graph convolutional layer cascaded sequentially; the second convolutional layer has a kernel size of 1×1, a stride of (1,1), and padding of (0,0); the TCN block is composed of a batch normalization layer, a ReLU activation function, a third convolutional layer, a batch normalization layer, and a Dropout layer cascaded sequentially; the third convolutional layer has a kernel size of 75×1, a stride of (1,1), and padding of (37,0); the input data D2 of the residual ST-GCN block first passes through a GCN block and a TCN block sequentially to output a feature vector E4; the feature vector E4 passes through a ReLU activation function to output a feature vector E5; the feature vector E4 then passes through the residual layer to output a feature vector E6; the feature vectors E5 and E6 are added to output a feature vector D3.

[0088] The training samples of the MSMTA-GCN network are in the form of (J F J S B F B S Data for training samples, where Y represents the true class label of the training samples;

[0089] In each iteration of the MSMTA-GCN network, the input training sample data of size B×C×T×V first passes through an MTA-GCN block to obtain a feature vector D1 of size B×C1×T×V; feature vector D1 passes through a ReLU activation function to obtain a feature vector D2 of size B×C1×T×V; feature vector D2 passes through a residual ST-GCN block to obtain a feature vector of size B×C1×T×V. The feature vector D3; the feature vector D3 is processed through an ST-GCN block to obtain a size of The feature vector D4 is processed by an average pooling layer to obtain a feature vector D5 of size B×C2×1×1. Finally, the feature vector D5 is processed by a convolutional layer with a kernel size of 1×1 and a stride of (1,1) to obtain a probability distribution P of the predicted emotion category of size B×4×1×1. The probability values ​​of belonging to the same emotion category are summed and normalized by softmax. The emotion category with the highest probability value is taken as the final predicted emotion category of the network. The classification loss in the form of cross-entropy is calculated based on the true category label Y of the training samples and the probability distribution P of the predicted emotion category to update the network parameters of the MSMTA-GCN network.

[0090] The classification loss has the following functional form:

[0091]

[0092] Where N is the number of categories of human emotions, p n and y n These are the nth-dimensional elements of P and Y, respectively.

[0093] The training set of the MSMTA-GCN network is an emotion gait dataset, which contains 2177 joint movement sequences of real human gait, with human emotion categories of happiness, sadness, anger, or indifference.

[0094] The MSMTA-GCN network uses five-fold cross-validation, with a training sample data to test sample data ratio of 4:1.

[0095] In each iteration, the training samples of the MSMTA-GCN network are processed in batches, with each batch containing 8 training samples.

[0096] The MSMTA-GCN network is set to iterate 200 times, and the initial learning rate of the MSMTA-GCN network is set to 0.01. Starting from the 100th iteration, the initial learning rate is divided by 10 every 50 iterations until 200 iterations are completed.

[0097] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. An emotion recognition method based on 3D skeletal gait analysis, characterized in that, Includes the following steps: S1. The joint motion sequence of human gait in walking video is used as raw data. After data preprocessing, the raw data is used to obtain local frame stream of joint information, global sequence stream of joint information, local frame stream of skeleton information, and global sequence stream of skeleton information, forming a fused multi-stream structure. S2. Obtain a pre-trained MSMTA-GCN network, wherein the MSMTA-GCN network structure is composed of an MTA-GCN block, a ReLU activation function, a residual ST-GCN block, an ST-GCN block, an average pooling layer, and a convolutional layer cascaded in sequence; S3. The local frame stream of joint information, the global sequence stream of joint information, the local frame stream of skeleton information, and the global sequence stream of skeleton information are respectively used as inputs to the MSMTA-GCN network to obtain the probability distribution of emotion prediction category under each input condition. The probability values ​​belonging to the same emotion prediction category are added together to form a new probability distribution. The new probability distribution is normalized by softmax, and the emotion prediction category with the highest probability value is taken as the emotion category finally predicted by the network to realize the recognition of human emotions. In step S2, the input data of the MTA-GCN block first passes through an attention module and a batch normalization layer, and outputs a feature vector. The input data of the MTA-GCN block is then passed through the residual layer to output a feature vector. ,Will and The eigenvectors are obtained by adding them together. ,Will The output feature vector is generated after passing through a ReLU activation function, a batch normalization layer, another ReLU activation function, a first convolutional layer, another batch normalization layer, and a Dropout layer. The residual layer consists of a convolutional layer with a kernel size of 1×1 and a stride of (1,1) and a batch normalization layer cascaded together; the first convolutional layer has a kernel size of 75×1, a stride of (1,1), and padding of (37,0); the attention module includes parallel... Threads Input data The input is fed into the attention module to obtain the output. The function form is: in, The kernel size represents the spatial dimension; express The k-th weight matrix in the middle; It is the adjacency matrix of the original normalized human physical structure, and The element value in each The middle remains unchanged. ; express The k-th adjacency matrix in the training process; The element values ​​are parameterized and optimized; It is a data correlation graph uniquely learned and determined for each sample. The function form is: Where G represents a graph, and This represents two different embedding functions, both of which are 1×1 convolutional layers, but with different strides and padding methods. The residual layer function is as follows: in, express The weight matrix; Batch normalization layer; S2. Obtain a pre-trained MSMTA-GCN network, wherein the MSMTA-GCN network structure is composed of an MTA-GCN block, a ReLU activation function, a residual ST-GCN block, an ST-GCN block, an average pooling layer, and a convolutional layer cascaded in sequence.

2. The emotion recognition method based on 3D skeletal gait analysis as described in claim 1, characterized in that, The data preprocessing method in step S1 is as follows: S11. Establish a camera coordinate system centered on the camera and a human coordinate system centered on the human body. In the camera coordinate system, the origin is the optical center of the camera, the X-axis is parallel to the long side of the camera screen, the Y-axis is parallel to the short side of the camera screen, and the Z-axis is the optical axis of the camera. In the human coordinate system, the origin is the center of gravity of the human body, the X-axis is the shoulder, and the Y-axis is the spine. S12. Acquire a video of a human walking, and continuously extract H frames from the video as a sequence of joint movements in the human gait. When the number of video frames is less than H frames, it is expanded to H frames by looping the video and connecting the beginning and end; in each frame, the body faces the same direction; joint movement sequence. All joints reference the same camera coordinate system and the same human coordinate system; S13. From joint movement sequence Local frame stream for obtaining joint information , The t-th time frame in Joint The function form is: in, Indicates the center of gravity of the human body. It is a joint movement sequence The t-th time frame One joint, It is the center joint of the t-th time frame. It is the rotation matrix of the t-th time frame. The function form is: in, It is the angle between the X-axis of the camera coordinate system and the X-axis of the human body coordinate system. It is the angle between the Z-axis of the camera coordinate system and the y-axis of the human coordinate system. It is based on The counterclockwise rotation matrix of the X-axis in the radian rotation camera coordinate system. It is based on The counterclockwise rotation matrix of the Z-axis in the radian rotation camera coordinate system; S14. From joint movement sequence Obtain the global sequence stream of joint information , The t-th time frame in Joint The function form is in, It is the center joint of the 0th time frame; S15. Local frame stream based on joint information Local frame stream that obtains skeletal information , The t-th time frame in The function form is: in, It is the time frame t that is related to the time frame t. The joints adjacent to each other; S16. Global sequence stream of joint information Obtain the global sequence stream of skeletal information , The t-th time frame in The function form is:

3. The emotion recognition method based on 3D skeletal gait analysis as described in claim 1, characterized in that, In the MSMTA-GCN network, the ST-GCN block is composed of a GCN block, a TCN block, and a ReLU activation function cascaded sequentially; the GCN block is composed of a second convolutional layer and a graph convolutional layer cascaded sequentially; the second convolutional layer has a kernel size of 1×1, a stride of (1,1), and padding of (0,0); the TCN block is composed of a batch normalization layer, a ReLU activation function, a third convolutional layer, a batch normalization layer, and a Dropout layer cascaded sequentially; the third convolutional layer has a kernel size of 75×1, a stride of (1,1), and padding of (37,0); the input data of the residual ST-GCN block... First, the feature vector is output after passing through a GCN block and then a TCN block. ; Eigenvector The feature vector is output after the ReLU activation function. ; Eigenvector The feature vector is then output through the residual layer. ; to feature vector With feature vectors Add to output feature vector .

4. The emotion recognition method based on 3D skeletal gait analysis as described in claim 3, characterized in that, The training samples for the MSMTA-GCN network are in the form of ( Data, where Y represents the true class label of the training samples; In each iteration of the MSMTA-GCN network, the input training sample data of size B×C×T×V is first processed through an MTA-GCN block to obtain a sample of size B× The eigenvector of ×T×V ; Eigenvector After the ReLU activation function, the resulting size is B× The eigenvector of ×T×V ; Eigenvector After passing through the residual ST-GCN block, a size of B× is obtained. × The eigenvector of ×V ; Eigenvector After passing through the ST-GCN block, a size of B× is obtained. × The eigenvector of ×V ; to feature vector After the average pooling layer, a size of B× is obtained. × eigenvector of ×1 ; final eigenvector After passing through a convolutional layer with a kernel size of 1×1 and a stride of (1,1), a result of size B× is obtained. × The network parameters of the MSMTA-GCN network are updated by calculating the cross-entropy classification loss based on the true class label Y of the training samples and the probability distribution of the predicted emotion category P, summing the probability values ​​belonging to the same emotion category, and normalizing them by softmax. The emotion category with the highest probability value is taken as the final predicted emotion category of the network. The network parameters of the MSMTA-GCN network are updated by calculating the cross-entropy classification loss based on the true class label Y of the training samples and the probability distribution of the predicted emotion category P.

5. The emotion recognition method based on 3D skeletal gait analysis as described in claim 4, characterized in that, The classification loss has the following functional form: in, It is the number of categories of human emotions. , , and They are respectively and The Dimensional elements.

6. The emotion recognition method based on 3D skeletal gait analysis as described in claim 5, characterized in that, The training set of the MSMTA-GCN network is an emotion gait dataset, which contains 2177 joint movement sequences of real human gait, with human emotion categories of happiness, sadness, anger, or indifference.

7. The emotion recognition method based on 3D skeletal gait analysis as described in claim 5, characterized in that, The MSMTA-GCN network uses five-fold cross-validation, with a training sample data to test sample data ratio of 4:

1.

8. The emotion recognition method based on 3D skeletal gait analysis as described in claim 5, characterized in that, In each iteration, the training samples of the MSMTA-GCN network are processed in batches, with each batch containing 8 training samples.

9. The emotion recognition method based on 3D skeletal gait analysis as described in claim 5, characterized in that, The MSMTA-GCN network is set to iterate 200 times, and the initial learning rate of the MSMTA-GCN network is set to 0.

01. Starting from the 100th iteration, the initial learning rate is divided by 10 every 50 iterations until 200 iterations are completed.

Images