A limb emotion classification method based on a graph convolution network

Abstract

The application discloses a limb emotion classification method based on a graph convolution network, belongs to the field of machine learning classification, and aims to solve the problem of effectively extracting space-time feature information in data for emotion classification of limb actions with large intra-class variance and small inter-class variance, and effectively extracting discriminative features, points are as follows: converting bone point action data of a target object into a first matrix; inputting the first matrix into a channel topology graph convolution module to obtain a channel space topology feature matrix; inputting the space topology feature matrix into a multi-scale time convolution module to obtain a channel topology space-time feature matrix; performing global average pooling on the channel topology space-time feature matrix; and classifying the channel topology space-time feature matrix after the global average pooling through a classifier to obtain limb emotion classification, so that the precision of emotion classification for limb actions is effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of machine learning classification, and relates to a limb emotion classification method based on a graph convolution network. BACKGROUND

[0002] Emotion plays a crucial role in the process of decision-making, perception and learning of human beings, and has a significant impact on the rational thinking mechanism of human beings. One of the biggest differences between computers and human brains is that computers are difficult to understand and distinguish human emotions. If computers are to truly interact with humans naturally, computers need to be given the ability to recognize and understand emotions. However, due to technical limitations, previous control systems often do not have user emotion recognition, resulting in poor user experience in the process of human-computer interaction.

[0003] At present, with the continuous improvement of people's pursuit of human-computer interaction experience, emotion recognition technology has gradually become an important research field in pattern recognition, and therefore emotion recognition technology has been widely used in important application scenarios such as online education and public security. Humans can judge each other's emotions through facial expressions, voice tones and body movements. Mehrabian formulated the 7-38-55 rule to better analyze human emotions. The rule states that human emotional information is composed of 7% language and conversation content, 38% speaking tone and tone of voice, and 55% body movement and facial expression. It can be seen that body movement plays a very important role in emotion expression and understanding. In today's society, people have higher and higher requirements for privacy protection, and in most cases do not want their conversation content and appearance characteristics to be monitored. However, people are more accepting of the use of body movement data. Therefore, in application fields such as public place behavior detection and satisfaction survey, emotion classification technology based on body movement has a larger and freer application scenario and development prospect.

[0004] Most emotion recognition methods based on machine learning methods tend to study a specific set of motion features. For example, Glowinski et al. adopted the spatial range, smoothness and symmetry of body movements as relevant features for main discrimination, and then used principal component analysis to realize the final emotion classification. However, the emotion classification method based on machine learning method proposed by Glowinski et al. is limited by the model, which can confuse the features between different emotions and is difficult to extract effective discrimination information. The classification accuracy of such emotion classification method is generally low, and the main reason is that traditional machine learning models are difficult to effectively extract features in data with large intra-class variance and small inter-class variance.

[0005] In recent years, researchers have begun to apply deep learning methods to related fields. Yan et al. proposed STGCN, which extracts human limb action features in the time dimension and the space dimension respectively, and constructs a spatio-temporal graph convolutional neural network. However, the deep learning method STGCN proposed by Yan et al. artificially divides the human skeleton points into fixed subsets and manually defines the unchangeable adjacency matrix, which limits the extraction ability of the graph convolutional neural network for human skeleton point feature information. SUMMARY

[0006] In order to solve the problem of effectively extracting spatio-temporal feature information in data for limb action emotion classification with large intra-class variance and small inter-class variance, and to effectively extract discriminative features, in a first aspect, a limb emotion classification method based on a graph convolutional network according to some embodiments of the present application comprises

[0007] S10. converting the skeleton point action data of the target object into a first matrix, the first matrix comprising the number of skeleton points of the object, the feature dimension of the skeleton point data, and the frame number of the skeleton point data;

[0008] S20. inputting the first matrix into a channel topology graph convolution module to obtain a channel spatial topology feature matrix;

[0009] S30. inputting the spatial topology feature matrix into a multi-scale temporal convolution module to obtain a channel topology spatio-temporal feature matrix;

[0010] S40. performing global average pooling on the channel topology spatio-temporal feature matrix;

[0011] S50. classifying the channel topology spatio-temporal feature matrix after global average pooling by a classifier to obtain limb emotion classification.

[0012] The limb emotion classification method based on the graph convolutional network according to some embodiments of the present application is executed by a classification network, and the classification network comprises

[0013] a channel topology spatio-temporal feature extraction layer comprising a channel topology graph convolution module and a multi-scale temporal convolution module, wherein the channel topology graph convolution module inputs a first matrix and outputs a spatial topology feature matrix, and the multi-scale temporal convolution module inputs the spatial topology feature matrix and outputs a channel topology spatio-temporal feature matrix;

[0014] a global average pooling layer for performing global average pooling on the channel topology spatio-temporal feature matrix;

[0015] a SoftMax classifier for classifying the channel topology spatio-temporal feature matrix after global average pooling by the SoftMax classifier to obtain limb emotion classification.

[0016] According to some embodiments of the limb emotion classification method based on graph convolutional networks in this application, the channel topology graph convolutional module obtains the channel space topology feature matrix in the following manner:

[0017] S201. Construct the first adjacency matrix between the skeleton points;

[0018] S202. The first matrix is ​​refined using a feature refinement function to obtain a higher-order feature matrix;

[0019] S203. Compress the time dimension and refine the feature dimension of the first matrix using different linear transformation functions to obtain the second matrix and the third matrix respectively;

[0020] S204. Obtain the second adjacency matrix of each channel of the first matrix, and obtain the second adjacency matrix of each channel through the channel topology function;

[0021] S205. The first adjacency matrix and the second adjacency matrix are fused using a fusion function to obtain the channel topology adjacency matrix;

[0022] S206. The higher-order feature matrix is ​​fused with the channel topology adjacency matrix using a fusion function to obtain the channel topology feature matrix;

[0023] S207. Repeat steps S201 to S207 to obtain two or more channel topology feature matrices. Then, fuse the two or more channel topology feature matrices using a fusion function to obtain the channel space topology feature matrix.

[0024] According to some embodiments of the present application, the body emotion classification method based on graph convolutional networks, the first matrix X is a V×C×T matrix, where V represents the number of skeletal points of the object, C represents the feature dimension of the skeletal point data, and T represents the number of frames of the acquired skeletal point data. In the initial state, C=3.

[0025] in:

[0026] S201. Construct the first adjacency matrix between skeleton points.

[0027] S202. By refining the feature function The first matrix Refine the first feature dimension to obtain the higher-order feature matrix X. ′ It can be expressed by formula (1):

[0028]

[0029] in, Let C represent the optimizable weight matrix. ′a feature dimension of the first feature dimension-refined skeletal point data;

[0030] S203. The first matrix is compressed in time dimension and refined in second feature dimension by different linear change functions φ(·), ψ(·) to obtain a second matrix a third matrix which are represented by formulas (2)-(3):

[0031] X1=φ(X)=meanT(XW1) (2)

[0032] X2=ψ(X)=meanT(XW2) (3)

[0033] wherein, W1, W2 represent weight matrices to be optimized, meanT(·) represents an average value along the time dimension, C 0 a feature dimension of the second feature dimension-refined skeletal point data;

[0034] S204. The first matrix is compressed in time dimension and refined in second feature dimension by different linear change functions φ(·), ψ(·) to obtain a second matrix a second adjacency matrix of each channel by a channel topology function a second adjacency matrix of each channel which is represented by formula (4):

[0035]

[0036] wherein, the channel topology function the subtraction in the formula (4) is a broadcast subtraction, and σ(·) represents an activation function, W1, W2 represent weight matrices to be optimized;

[0037] S205. A fusion function is used to fuse the first adjacency matrix and the second adjacency matrix containing the topology relationship of each channel to obtain a channel topology adjacency matrix which is represented by formula (5):

[0038]

[0039] wherein, the fusion function the addition in the formula (5) is a broadcast addition, and α represents a scalar to be optimized.

[0040] S206. A fusion function is used to fuse the high-order feature matrix and the channel topology adjacency matrix ​fusion, obtain a channel topology feature matrix is represented by formula (6):

[0041]

[0042] wherein || represents concatenating matrices in the feature dimension, P C′ represents the C'th VxV channel topology adjacency matrix sliced out from P of VxVxC', X' C′ represents the C'th CxT high-order feature matrix sliced out from X' of VxC'T.

[0043] S207. Repeat steps S201-S207 three times to obtain three channel topology feature matrices Z1, Z2 and Z3, and fuse the three channel topology feature matrices Z1, Z2 and Z3 through a fusion function to obtain a channel spatial topology feature matrix Z. is represented by formula (7):

[0044]

[0045] wherein σ(·) represents an activation function.

[0046] According to the limb emotion classification method based on the graph convolution network according to some embodiments of the present application, the multi-scale time convolution module obtains a channel topology space-time feature matrix based on the following manner:

[0047] S301. The channel spatial topology feature matrix is input into two scale time convolution modules respectively.

[0048] S302. The outputs of the two scale time convolution modules are added.

[0049] S303. The addition result is activated through an activation function to obtain a channel topology space-time feature matrix.

[0050] According to the limb emotion classification method based on the graph convolution network according to some embodiments of the present application, wherein: the channel topology space-time feature matrix is represented by formula (8)-(10):

[0051]

[0052]

[0053]

[0054] wherein DilatedConv(·) represents a dilated convolution function, MaxPool(·) represents a maximum pooling function, The convolution kernel size of DilatedConv(·) in the equation is 5, and the dilation coefficients are 2 and 3, respectively, The convolution kernel size of DilatedConv(·) in the equation is 3, and the dilation coefficients are 2 and 3, respectively, and W4, W5, W6, W7, W8 and W9 represent the weight matrices that can be optimized, respectively.

[0055] According to the limb emotion classification method based on the graph convolution network, the channel topology spatio-temporal feature extraction layer includes i+1 layers, each layer includes a channel topology graph convolution module and a scale time convolution module, wherein the specific method for obtaining the channel topology spatio-temporal feature matrix in steps S20-S30 comprises

[0056] The first matrix is input into the channel topology graph convolution module of the first layer channel topology spatio-temporal feature extraction layer, and the channel topology graph convolution module of the first layer channel topology spatio-temporal feature extraction layer outputs a first channel spatial topology feature matrix;

[0057] The first channel spatial topology feature matrix is input into the multi-scale time convolution module of the first layer channel topology spatio-temporal feature extraction layer, and the multi-scale time convolution module of the first layer channel topology spatio-temporal feature extraction layer outputs a first channel topology spatio-temporal feature matrix;

[0058] The first channel topology spatio-temporal feature matrix is input into the channel topology graph convolution module of the second layer channel topology spatio-temporal feature extraction layer, and the channel topology graph convolution module of the second layer channel topology spatio-temporal feature extraction layer outputs a second channel spatial topology feature matrix;

[0059] The second channel spatial topology feature matrix is input into the multi-scale time convolution module of the second layer channel topology spatio-temporal feature extraction layer, and the multi-scale time convolution module of the second layer channel topology spatio-temporal feature extraction layer outputs a second channel topology spatio-temporal feature matrix; ...

[0061] The i-1 channel topology spatio-temporal feature matrix is input into the channel topology graph convolution module of the i layer channel topology spatio-temporal feature extraction layer, and the channel topology graph convolution module of the i layer channel topology spatio-temporal feature extraction layer outputs an i channel spatial topology feature matrix;

[0062] The i channel spatial topology feature matrix is input into the multi-scale time convolution module of the i layer channel topology spatio-temporal feature extraction layer, and the multi-scale time convolution module of the i layer channel topology spatio-temporal feature extraction layer outputs an i channel topology spatio-temporal feature matrix;

[0063] input the i-th channel topological spatio-temporal feature matrix into a channel topological graph convolution module of an i+1-th layer channel topological spatio-temporal feature extraction layer, and the channel topological graph convolution module of the i+1-th layer channel topological spatio-temporal feature extraction layer outputs an i+1-th channel spatial topological feature matrix;

[0064] The i+1-th channel spatial topological feature matrix is input into a multi-scale time convolution module of the i+1-th layer channel topological spatio-temporal feature extraction layer, and the multi-scale time convolution module of the i+1-th layer channel topological spatio-temporal feature extraction layer outputs an i+1-th channel topological spatio-temporal feature matrix.

[0065] S40. The i+1-th channel topological spatio-temporal feature matrix output by the multi-scale time convolution module of the i+1-th layer channel topological spatio-temporal feature extraction layer is globally averaged pooled.

[0066] S50. The channel topological spatio-temporal feature matrix after global average pooling is classified by a classifier to obtain limb emotion classification.

[0067] According to the limb emotion classification method based on the graph convolution network, the channel topological spatio-temporal feature extraction layer includes ten layers.

[0068] According to the limb emotion classification method based on the graph convolution network, the number of channels of the ten-layer channel topological spatio-temporal feature extraction layer is 64-64-64-64-128-128-128-256-256-256.

[0069] According to the limb emotion classification method based on the graph convolution network, in the classification network training, a cross-entropy loss function is used to calculate the gap between the classification result and the real classification result, and then the classification network is optimized by back propagation.

[0070] Beneficial effects:

[0071] In the first aspect, the application effectively improves the accuracy of emotion classification for limb actions by inputting the skeletal point action data representing human limb actions as input, and extracting spatial feature information and temporal feature information in the skeletal point action data, respectively.

[0072] In the second aspect, to effectively extract spatial feature information in the original data, the application constructs a channel topological graph convolution module, which can construct an adjacency matrix between skeletal points along the feature dimension. Compared with the existing method, the application can more effectively extract spatial feature information in the original data while maintaining the structural nature of the feature information.

[0073] In a third aspect, in order to effectively extract time information in the original data, the application constructs a multi-scale time convolution module, adopts a multi-scale method, and performs one-dimensional convolution on the extracted spatial feature information along the time dimension, so that the limited time information in the long video is extracted and effectively converted into classification information.

[0074] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and drawings or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 Channel topology graph convolution module structure diagram.

[0076] Figure 2 Multi-scale time convolution module structure diagram.

[0077] Figure 3 Network structure diagram of the limb emotion classification method based on the graph convolution network. DETAILED DESCRIPTION

[0078] The embodiments of the application are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, and the application provides a method to solve the problem of effectively extracting spatio-temporal feature information in data for emotion classification of limb action with large intra-class variance and small inter-class variance, and effectively extracting discriminative features. The application belongs to the field of graph convolution neural network limb emotion classification method, and the skeletal point action data expressing human limb action is input into the classification network in the form of a matrix, a plurality of dynamic adjacency matrices are constructed along the feature dimension of the human skeletal point data by the channel topology graph convolution module, the human limb data is subjected to graph convolution operation using the obtained dynamic adjacency matrix, effective spatial feature information is extracted, then the multi-scale time convolution module is used to perform convolution operation on the obtained spatial feature information along the time dimension, channel topology spatio-temporal feature information is obtained, and based on the channel topology spatio-temporal feature information, the emotion classification expressed by the skeletal point action data is obtained through the pooling layer and the classification layer.

[0079] In one embodiment, the present disclosure describes a limb emotion classification method based on a graph convolution network, and the method uses a classification network to perform emotion classification. In this embodiment, the composition, training and use of the classification network are involved.

[0080] The classification network comprises a channel topology spatiotemporal feature extraction layer, a global average pooling layer and a SoftMax classifier, wherein the channel topology spatiotemporal feature extraction layer comprises a channel topology graph convolution module and the multi-scale time convolution module, wherein the channel topology graph convolution module inputs a first matrix and outputs a spatial topology feature matrix, and the multi-scale time convolution module inputs the spatial topology feature matrix and outputs a channel topology spatiotemporal feature matrix. The global average pooling layer is used for globally average pooling the channel topology spatiotemporal feature matrix; and the SoftMax classifier is used for limb emotion classification of the channel topology spatiotemporal feature matrix after global average pooling through the SoftMax classifier.

[0081] The limb emotion classification method based on the graph convolution network comprises the following steps:

[0082] (1) A sensor such as a depth camera is used to obtain bone point action data of a target object.

[0083] (2) The obtained bone point action data is sorted and converted into a matrix X of VxCXT, wherein V represents the number of bone points of the object, C represents the feature dimension of the bone point data, C=3 in an initial state, and T represents the frame number of the obtained bone point data.

[0084] As shown in the formula (1), the channel topology graph convolution module performs the following specific steps: Figure 1

[0085] (3) Priori knowledge is used to manually construct an adjacency matrix between V bone points, and an overall adjacency matrix A is initialized.

[0086] (4) The matrix X converted from the original bone point data is subjected to first feature dimension refinement, and the matrix X is converted into a high-order feature matrix X by a feature refinement function

[0087]

[0088] wherein W is an optimizable weight matrix, and C represents the feature dimension of the bone point data after the first feature dimension refinement. ′

[0089] (5) The matrix X converted from the original bone point data is compressed in the time dimension and refined in the feature dimension (second feature dimension refinement), and the matrix X is converted into and by using a linear change function φ(·) and ψ(·) respectively.

[0090] ​​​​​​X1=φ(X)=meanT(XW1) (2)

[0091] X2=ψ(X)=meanT(XW2) (3)

[0092] in, and For an optimizable weight matrix, the meanT(·) function represents the average value along the time dimension, C 0 This represents the feature dimension of the skeletal point data after refinement in the second feature dimension.

[0093] (6) Obtain the matrix X containing the skeletal point data and the adjacency matrix for each channel. Through channel topology functions Calculate the topological relationships on each channel, expressed by the adjacency matrix Q, using X1 and X2:

[0094]

[0095] Where the channel topology function The subtraction method used is broadcast subtraction, and σ(·) is the activation function. This is an optimizable weight matrix.

[0096] (7) Initialize the global adjacency matrix Adjacency matrix containing the topological relationships of each channel The fusion process yields a channel topology adjacency matrix that simultaneously possesses shared topology feature information and channel internal topology feature information. Using fusion function Finish:

[0097] (5)

[0098] Where the fusion function The addition in the expression is broadcast addition, and α is an optimizable scalar.

[0099] Typically, existing adjacency matrices are V×V, lacking information about the internal topological features of channels. Such adjacency matrices contain limited spatial information, resulting in insufficient spatial data acquisition. The adjacency matrix Q of this invention is constructed as V×V×C. ′ This involves constructing multiple dynamic adjacency matrices along the feature dimensions of human skeletal point data, which can contain more spatial information and facilitate deeper extraction of spatial information. On this basis, fusion is required so that the channel topological adjacency matrix P can simultaneously possess shared topological feature information and channel internal topological feature information. These two types of information are used to make the topological adjacency matrix P have more spatial information, which is beneficial for further deeper extraction of spatial information.

[0100] According to the scheme, the channel topology adjacency matrix P has both the channel internal topology feature information and the shared topology feature information,

[0101] (8) The high-order feature matrix and the channel topology adjacency matrix are fused by a feature fusion function to obtain the channel topology feature matrix

[0102]

[0103] Where || represents concatenation of matrices in the feature dimension, P C′ represents the C'th VxV channel topology adjacency matrix sliced out of P of VxVxC', and X' C′ represents the C'th VxT high-order feature matrix sliced out of X' of VxC'xT.

[0104] (9) Repeat steps (3) to (8) three times to obtain Z1, Z2 and Z3 respectively, and then perform feature fusion of Z1, Z2 and Z3 by a feature fusion function to obtain the channel spatial topology feature matrix

[0105]

[0106] As shown in Figure 2 , the multi-scale time convolution module performs the following specific steps:

[0107] (10) The channel spatial topology feature matrix is input into two time convolution modules with convolution kernel sizes of 3 and 5 respectively as a multi-scale time convolution module, and then the outputs of the two scales of time convolution modules are added and activated to obtain the output result of the multi-scale time convolution module

[0108]

[0109]

[0110]

[0111] Where DilatedConv(·) is a dilated convolution function, MaxPool(·) is a max-pooling function, DilatedConv(·) in The DilatedConv(·) function uses a convolution kernel size of 3, with dilation coefficients of 2 and 3, and W4, W5, W6, W7, W8, and W9 are optimizable weight matrices.

[0112] This invention defines two different scales, enabling the channel topology spatiotemporal feature matrix S to simultaneously contain small-scale local feature information and large-scale global feature information. This balances detailed and global information, which is beneficial for mining feature correlations and extracting temporal information more comprehensively, thereby improving accuracy.

[0113] In a preferred example, such as Figure 3 As shown, the entire classification network structure consists of 10 channel topology spatiotemporal feature extraction layers, one global average pooling layer, and a SoftMax classifier. Its specific implementation includes the following steps:

[0114] (11) The single-layer channel topology spatiotemporal feature extraction layer obtains channel topology spatiotemporal feature information through steps (1) to (10). The number of output feature channels of the 10-layer channel topology spatiotemporal feature extraction layer are 64-64-64-64-128-128-128-256-256-256.

[0115] To obtain the channel topology spatiotemporal feature information using 10 extraction layers, the following method is employed:

[0116] The first matrix is ​​input into the channel topology graph convolution module of the first channel topology spatiotemporal feature extraction layer, and the channel topology graph convolution module of the first channel topology spatiotemporal feature extraction layer outputs the first channel spatial topology feature matrix.

[0117] The first channel spatial topological feature matrix is ​​input into the multi-scale temporal convolution module of the first channel topological spatiotemporal feature extraction layer, and the multi-scale temporal convolution module of the first channel topological spatiotemporal feature extraction layer outputs the first channel topological spatiotemporal feature matrix.

[0118] The first channel topological spatiotemporal feature matrix is ​​input into the channel topology graph convolution module of the second channel topological spatiotemporal feature extraction layer, and the channel topology graph convolution module of the second channel topological spatiotemporal feature extraction layer outputs the second channel spatial topology feature matrix.

[0119] The second channel spatial topological feature matrix is ​​input into the multi-scale temporal convolution module of the second-layer channel topological spatiotemporal feature extraction layer, and the multi-scale temporal convolution module of the second-layer channel topological spatiotemporal feature extraction layer outputs the second channel topological spatiotemporal feature matrix.

[0120] ...

[0121] The eighth channel topology space-time feature matrix is input into a channel topology graph convolution module of a ninth layer channel topology space-time feature extraction layer, and the channel topology graph convolution module of the ninth layer channel topology space-time feature extraction layer outputs a ninth channel topology space-time feature matrix.

[0122] The ninth channel topology space-time feature matrix is input into a multi-scale time convolution module of the ninth layer channel topology space-time feature extraction layer, and the multi-scale time convolution module of the ninth layer channel topology space-time feature extraction layer outputs a ninth channel topology space-time feature matrix.

[0123] The ninth channel topology space-time feature matrix is input into a channel topology graph convolution module of a tenth layer channel topology space-time feature extraction layer, and the channel topology graph convolution module of the tenth layer channel topology space-time feature extraction layer outputs a tenth channel topology space-time feature matrix.

[0124] The tenth channel topology space-time feature matrix is input into a multi-scale time convolution module of the tenth layer channel topology space-time feature extraction layer, and the multi-scale time convolution module of the tenth layer channel topology space-time feature extraction layer outputs a tenth channel topology space-time feature matrix.

[0125] According to the scheme, the extraction accuracy can be improved by selecting multiple layers.

[0126] In this scheme, the tenth channel topology space-time feature matrix output by the multi-scale time convolution module of the tenth layer channel topology space-time feature extraction layer is globally averaged. The channel topology space-time feature matrix after global average pooling is classified by a classifier to obtain limb emotion classification.

[0127] In an example, the classification network training method is as follows:

[0128] (1) In this example, the skeleton point data in the video-emotion data set is divided into a training set T1 and a test set T2.

[0129] (2) The skeleton point data in the training set T1 is input into the channel topology graph convolution module constructed above in small batches to extract channel topology space feature information.

[0130] (3) The obtained channel topology space feature information is input into the multi-scale time convolution module constructed above to extract channel topology space-time feature information.

[0131] (4) Repeat steps (2) and (3) 10 times, and adjust the number of feature channels output by each time to be 64-64-64-64-128-128-128-256-256-256, thereby completing the extraction of deep space-time feature information.

[0132] (5) The spatio-temporal feature information obtained in step (4) is converted into a classification result by using a 1-layer global average pooling layer and a SoftMax classifier.

[0133] (6) The difference between the classification result obtained by using a cross-entropy loss function and the real classification result is calculated, and then back propagation is performed to optimize the classification network parameters.

[0134] (7) Steps (2) to (6) are repeated until the classification result is close enough to the real classification result.

[0135] To further show that the present application has better effect compared with the existing emotion classification technology and the existing deep emotion classification technology, the technical scheme of the present application respectively constructs a channel topology graph convolution module and a multi-scale time convolution module, effectively extracts the spatial feature data and the time feature data in the target limb motion information, and greatly improves the emotion classification accuracy.

[0136] In the experimental example, the method of the present application is compared with other methods on the video-emotion data set provided by Dalian University of Technology. The data set contains 1406 data samples, which are composed of action videos recorded by 24 actors according to specific emotions. The data samples in the data set contain 7 categories, each category corresponds to an emotion, and the emotions are joy, anger, sadness, fear, dislike, surprise and neutral respectively. Since different people express emotions in different ways, the difference in actions exhibited by different performers when performing the same emotion is relatively large, and sometimes the emotions expressed by similar actions are different, which makes the data set have the characteristics of large intra-class variance and small inter-class variance, which greatly increases the classification difficulty of the emotion classification method under the data set. The experimental results of the method of the present application compared with other methods on the video-emotion data set are shown in Table 1.

[0137] Table 1 Test results of the method of the present application and other comparative algorithms

[0138] Algorithm Accuracy Algorithm 1 47.57％ Algorithm 2 62.23％ The invention 64.57％

[0139] Appendix: Algorithm 1: ST-GCN; Algorithm 2: CTR-GCN.

[0140] The CTR-GCN can achieve a classification accuracy of 90.6% in a conventional action classification data set such as NTU 60, which is one of the most advanced action classification methods at present, but the classification accuracy under the data set is only 62.23%, which shows that the difficulty of the video-emotion data set is great. The classification accuracy of the limb emotion classification method adopted in the application under the data set is 64.57%, which exceeds the CTR-GCN, and the accuracy of the method of the application can be improved by 2.34% compared with the CTR-GCN, so that the classification accuracy of the limb emotion is greatly improved.

[0141] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A limb emotion classification method based on a graph convolution network, characterized in that, Comprising S10. converting the bone point action data of the target object into a first matrix, the first matrix comprising the number of bone points of the object, the feature dimension of the bone point data, and the frame number of the bone point data; S20. inputting the first matrix into a channel topology graph convolution module to obtain a channel spatial topology feature matrix; S30. inputting the spatial topology feature matrix into a multi-scale time convolution module to obtain a channel topology space-time feature matrix; S40. performing global average pooling on the channel topology space-time feature matrix; S50. classifying the channel topology space-time feature matrix after global average pooling through a classifier to obtain limb emotion classification; The first matrix is a matrix, represents the number of bone points of an object, represents the feature dimension of bone point data, represents the frame number of acquired bone point data, and in an initial state ; The channel topology graph convolution module obtains the channel spatial topology feature matrix based on the following manner: S201. Construct a first adjacency matrix between the skeletal points ; S202. passing the feature refinement function obtaining the first matrix performing first feature dimension refinement to obtain a high-order feature matrix which is represented by formula (1): （1） wherein, represents the weight matrix that can be optimized, represents the feature dimension of the first skeleton point data after the feature dimension refinement; S203. Through different linear change functions 、 The first matrix is compressed in the time dimension, and the second feature dimension is refined to obtain a second matrix and a third matrix , which are represented by formulas (2)-(3): （2） （3） wherein, denotes the weight matrix that can be optimized, denotes the function that takes the average along the time dimension, denotes the feature dimension of the second feature dimension-refined skeletal point data; S204. obtaining said first matrix a second adjacency matrix for each channel by a channel topology function obtaining a second adjacency matrix for each channel is represented by equation (4): （4） where the channel topology function The subtraction used in the middle of the equation is a broadcast subtraction, represents an activation function, represents an optimizable weight matrix; S205. fusing the first adjacency matrix and the second adjacency matrix to obtain a channel topology adjacency matrix having both the internal topology characteristic information and the shared topology characteristic information of the channels and the second adjacency matrix including the topology relationship of each channel fusing to obtain a channel topology adjacency matrix having both the internal topology characteristic information and the shared topology characteristic information of the channels which is represented by formula (5): （5） where the addition in the fusion function is a broadcast add, denotes a scalar that can be optimized. S206. Obtain the high-order feature matrix by fusing the high-order feature matrix and the channel topology feature matrix S206. Obtain the high-order feature matrix by fusing the high-order feature matrix and the channel topology feature matrix S206. Obtain the high-order feature matrix by fusing the high-order feature matrix and the channel topology feature matrix S206. Obtain the high-order feature matrix by fusing the high-order feature matrix and the channel topology feature matrix S206. Obtain the high-order feature matrix by fusing the high-order feature matrix and the channel topology feature matrix （6） wherein, represents concatenating the matrix in the feature dimension, represents slicing the th channel topology adjacency matrix from the th high-order feature matrix of represents slicing the th high-order feature matrix from the th channel topology adjacency matrix of S207. Repeat steps S201-S207 three times to obtain three channel topology feature matrices , and Fuse the channel topology feature matrices , and by a fusion function to obtain a channel spatial topology feature matrix , which is represented by equation (7): （7） wherein, denotes an activation function. 2.The limb emotion classification method based on graph convolution network according to claim 1, characterized in that, The classification method is executed by a classification network, and the classification network comprises The channel topology space-time feature extraction layer comprises an i+1 layer, and each layer comprises a channel topology graph convolution module and a scale time convolution module, wherein the specific method for obtaining the channel topology space-time feature matrix in steps S20-S30 comprises inputting the first matrix into the channel topology graph convolution module of the first layer channel topology space-time feature extraction layer, and the channel topology graph convolution module of the first layer channel topology space-time feature extraction layer outputting a first channel spatial topology feature matrix; inputting the first channel spatial topology feature matrix into the multi-scale time convolution module of the first layer channel topology space-time feature extraction layer, and the multi-scale time convolution module of the first layer channel topology space-time feature extraction layer outputting a first channel topology space-time feature matrix; 3.The limb emotion classification method based on graph convolution network according to claim 1, characterized in that, inputting the first channel topology space-time feature matrix into the channel topology graph convolution module of the second layer channel topology space-time feature extraction layer, and the channel topology graph convolution module of the second layer channel topology space-time feature extraction layer outputting a second channel spatial topology feature matrix; inputting the second channel spatial topology feature matrix into the multi-scale time convolution module of the second layer channel topology space-time feature extraction layer, and the multi-scale time convolution module of the second layer channel topology space-time feature extraction layer outputting a second channel topology space-time feature matrix; ​ ​ 4. The limb-based affective classification method based on graph convolution network according to claim 3, characterized in that, ​ The channel topology space-time feature matrix are represented by equations (8)-(10): （8） （9） （10） wherein, denotes a dilated convolution function, denotes a max-pooling function, the convolution kernel size of is 5 and the dilation coefficients are 2 and 3, respectively, the convolution kernel size of is 3 and the dilation coefficients are 2 and 3, respectively, and denote the weight matrices that can be optimized, respectively.​​​​ 5. The method of claim 4, wherein the graph convolutional network-based limb emotion classification method is characterized by, ​ ​ ​ ​ ​ …… The i-1 channel topology spatio-temporal feature matrix is input into a channel topology graph convolution module of an i layer channel topology spatio-temporal feature extraction layer, and the channel topology graph convolution module of the i layer channel topology spatio-temporal feature extraction layer outputs an i channel spatial topology feature matrix; The i channel spatial topology feature matrix is input into a multi-scale time convolution module of the i layer channel topology spatio-temporal feature extraction layer, and the multi-scale time convolution module of the i layer channel topology spatio-temporal feature extraction layer outputs an i channel topology spatio-temporal feature matrix; The i channel topology spatio-temporal feature matrix is input into a channel topology graph convolution module of an i+1 layer channel topology spatio-temporal feature extraction layer, and the channel topology graph convolution module of the i+1 layer channel topology spatio-temporal feature extraction layer outputs an i+1 channel spatial topology feature matrix; The i+1 channel spatial topology feature matrix is input into a multi-scale time convolution module of the i+1 layer channel topology spatio-temporal feature extraction layer, and the multi-scale time convolution module of the i+1 layer channel topology spatio-temporal feature extraction layer outputs an i+1 channel topology spatio-temporal feature matrix; S40. The i+1 channel topology spatio-temporal feature matrix output by the multi-scale time convolution module of the i+1 layer channel topology spatio-temporal feature extraction layer is globally averaged pooled; S50. The channel topology spatio-temporal feature matrix after global average pooling is classified by a classifier to obtain limb emotion classification.

6. The method of claim 5, wherein the graph convolutional network-based limb emotion classification method is characterized by, The channel topology spatio-temporal feature extraction layer includes ten layers.

7. The method of claim 6, wherein the graph convolutional network-based limb emotion classification method is characterized by, The number of channels of the ten-layer channel topology spatio-temporal feature extraction layer is 64-64-64-64-128-128-128-256-256-256. 8.The limb emotion classification method based on graph convolution network according to claim 5, characterized in that, In the classification network training, a cross-entropy loss function is used to calculate the difference between the classification result and the real classification result, and then the classification network is optimized by back propagation.

Images