Taijiquan action recognition method based on graph convolution and self-attention mechanism

Abstract

The invention discloses a shadowboxing action recognition method based on graph convolution and a self-attention mechanism, which comprises the following steps: firstly, extracting a human body skeleton sequence from a collected shadowboxing video by using a human body posture estimation algorithm, and then constructing a graph convolution adjacency matrix according to shadowboxing activity prior characteristics; a Tai Chi action recognition model based on graph convolution and a self-attention mechanism is constructed, skeleton data are combined through graph convolution, spatial features are extracted, time features are extracted through a multi-head self-attention mechanism in a self-attention layer, classification and calculation are completed through a full-connection layer to obtain an action classification result, and Tai Chi action classification recognition is completed. According to the method, recognition and classification of human body actions in a shadowboxing video are realized by using graph convolution and a self-attention mechanism, spatial correlation of each joint point is calculated by using a graph convolution method, aggregation information aggregation of end points with relatively long physical distances is accelerated, time correlation of each frame is calculated in parallel by using the self-attention mechanism, and the time correlation of each frame is calculated in parallel. And the precision and the stability of the algorithm are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of action recognition, and in particular relates to a Tai Chi action recognition method based on graph convolution and self-attention mechanism. Background Art

[0002] Stroke has a high morbidity and disability rate, becoming a leading cause of disability in China and globally. It can lead to motor dysfunction in patients, necessitating exercise rehabilitation training to restore their quality of life. Studies have shown that Tai Chi, a traditional Chinese martial art, can help improve limb balance and coordination in stroke patients, promote muscle strength recovery, and is an effective exercise rehabilitation training method. During Tai Chi rehabilitation training, problems such as irregular movements and poor limb coordination may occur, requiring analysis and guidance to improve rehabilitation outcomes. With the widespread application of deep learning and machine learning, research has been conducted using deep learning or machine learning algorithms to recognize human motion based on images or videos.

[0003] The document "Ye Songtao, Zhou Yangzheng, Fan Hongjie, et al. Human motion recognition integrating causal relationships and spatiotemporal graph convolutional networks. Computer Science, 2021, 48(S2): 130-5" proposes a motion recognition method that integrates causal relationships and spatiotemporal graph convolutional networks. The causal relationship between adjacent joint movements is determined based on human body mechanics, and edge weights are assigned to the joint connection graph of the graph convolutional network. The effectiveness of the proposed method is verified through simulation experiments on Tai Chi datasets, but this method does not take into account the requirements of Tai Chi for the coordination of various parts of the body. Patent CN117612250A proposes a method for motion recognition using infrared data, but this method ignores the movement characteristics of the coordinated action of the upper and lower limbs in Tai Chi moves.

[0004] In summary, in the existing technology, the action recognition method based on graph convolution is often based on the original human body physical structure topology, and does not fully consider the requirements of Tai Chi exercise for the coordination of various parts of the body. Due to the information transmission mechanism of the graph convolution method, the information aggregation of nodes with a physical distance such as the palms of the hands is slow, and the coordination of the two hands is an important element of Tai Chi; the existing method of extracting multi-frame human body time features through recurrent networks or convolution methods can only obtain contextual information through sequential processing, and cannot calculate the correlation between each frame in parallel. For Tai Chi exercises with continuous movements, it is difficult to effectively capture the action features. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes a Tai Chi motion recognition method based on graph convolution and self-attention mechanism. The graph convolution method is used to calculate the spatial correlation of each joint point, and the self-attention mechanism is used to calculate the temporal correlation of each frame, effectively capturing motion features and improving the accuracy of Tai Chi motion recognition.

[0006] The technical solution adopted by the present invention is: a Tai Chi action recognition method based on graph convolution and self-attention mechanism, the specific steps are as follows:

[0007] S1, collect videos of Tai Chi training process, and use human posture estimation algorithm to extract human skeleton sequence from the collected Tai Chi videos;

[0008] S2. Based on step S1, construct a graph convolution adjacency matrix according to the prior features of Tai Chi activities;

[0009] S3. Build a Tai Chi action recognition model based on graph convolution and self-attention mechanism;

[0010] The recognition network model includes: M layers of TC-GCN block, fully connected layer, softmax layer; and the structure of TC-GCNblock includes: 1*1 convolution layer, GCN graph convolution layer, self-attention layer, Res residual block.

[0011] S4. Based on the model built in step S3, the skeleton data is aggregated through graph convolution to extract spatial features, and then the temporal features are extracted through the multi-head self-attention mechanism in the self-attention layer;

[0012] S5. Based on step S4, complete the classification through the fully connected layer, calculate the action classification result, and complete the Tai Chi action classification and recognition.

[0013] Furthermore, the step S2 is specifically as follows:

[0014] S21, physical prior setting of human skeleton;

[0015] Set the adjacency matrix A according to the physical connection of each key point of the human body 1 , set the i-th key point v i and the jth key point v j If they are adjacent on the human body, then let A 1 (i,j)=1, otherwise A 1 (i,j)=0.

[0016] If no special graph structure design is made for adjacent nodes, use l i (v j ) represents the connection type classification when performing graph convolution on key points i and j. In this figure, l i (v j )=1,l j (v i )=1, indicating the first type of connection, that is, physical connection.

[0017] S22, prior setting of Tai Chi movement characteristics;

[0018] The adjacency matrix is ​​set up according to the characteristics and requirements of Tai Chi exercises.

[0019] (1) Gesture requirements;

[0020] Set the adjacency matrix A 2 Capturing the Tai Chi hand gestures requires corresponding features. Preserving the original physical connection between the hands and shoulders, the A between the corresponding nodes 2 (v i ,v j )=1, set the physically connected adjacent points to the same subset, l i (v j )=1,l j (v i )=1.

[0021] Establish left-right relative logical connections between the key points corresponding to the left and right palms, arms, and shoulders, and the value A of the adjacency matrix between the corresponding nodes 2 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2, indicating the second type of connection, that is, logical connection;

[0022] Among them, some Tai Chi exercises require the relative position of the palm and shoulder, so a connection is established between the palm and the key point above the shoulder. The value A of the adjacency matrix between the corresponding nodes 2 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2.

[0023] The connections of the remaining graph nodes in the adjacency matrix are set to 0.

[0024] (2) Leg posture requirements;

[0025] Set the adjacency matrix A 3 Capturing the leg posture of Tai Chi requires corresponding characteristics. The original physical connection from the toes to the thighs, the A between the corresponding nodes 3 (v i ,v j )=1, set the physically connected adjacent points to the same subset, l i (v j )=1,l j (v i )=1.

[0026] Establish left-right relative logical connections between the key points corresponding to the left and right ankles, knee joints and thigh hip joints, and the A between the corresponding nodes 3 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2.

[0027] The remaining graph structure connections are set to 0.

[0028] (3) Head posture requirements;

[0029] Set the adjacency matrix A 4 Capturing the head posture of Tai Chi requires corresponding features. Establish logical connections between binocular key points and palm key points, and the A between corresponding nodes 4 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2.

[0030] Use the shoulder position to determine the front of the human body posture, establish a left-right relative logical connection between the shoulder key points and the binocular key points, and the A between the corresponding nodes 4 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2.

[0031] The remaining graph structure connections are set to 0.

[0032] (4) body trunk rotation characteristics;

[0033] Set the adjacency matrix A 5 Capture the trunk rotation characteristics of Tai Chi. Obtain the trunk rotation characteristics through the relative movement of the center key point of the human pelvis and the shoulders and legs, establish a logical connection between the center key point of the human pelvis and the shoulder key points, and the A between the corresponding nodes 5 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j)=2,l j (v i )=2.

[0034] The physical connection between the central key point of the human pelvis and the hip joints of the legs is retained. 5 (v i ,v j )=1, set the physically connected adjacent points to the same subset, l i (v j )=1,l j (v i )=1.

[0035] The remaining graph structure connections are set to 0.

[0036] Furthermore, the step S4 is specifically as follows:

[0037] S41, aggregate skeleton data through graph convolution and extract spatial features;

[0038] The skeleton sequence data X obtained in step S1 and the multiple adjacency matrices obtained in steps S21 and S22 are respectively input into the M-layer TC-GCN block network structure of the recognition model in step S3.

[0039] Among them, the skeleton sequence data of the input m-th layer TC-GCN block is represented as X m [T,N,C m ], N represents the total number of human skeleton joints input, T represents the number of frames of the input skeleton sequence, and the optimal number of layers M and C of the TC-GCN block structure are determined according to the actual situation. m Represents the dimension of the data input to the m-th layer TC-GCN block structure, C1 represents the data dimension of the input to the first layer TC-GCNblock structure, that is, the original input data. The original input data is 3D skeleton data, then C1 = 3, and the data dimension of the input to the m-th layer TC-GCN block is the data dimension of the output of the m-1-th layer TC-GCN block.

[0040] Then, in the TC-GCN block network structure, a 1*1 convolution kernel is used to change the number of feature channels and divide the K skeleton posture multi-channel information into Input L layers of GCN to extract spatial information and use graph convolution to aggregate relevant skeleton node information on a single frame. The expression is as follows:

[0041]

[0042] Among them, l represents the lth layer of GCN, with a total of L layers; X (l)A represents the input of the first layer of the network, A represents the adjacency matrix used for graph convolution calculation, which is used to represent the connection relationship between each graph node, K represents the total number of adjacency matrices used, and K = 5. k represents the kth adjacency matrix (1≤k≤K), each adjacency matrix is ​​set according to step S2, Indicates A k The matrix after adding the self-loop, and I represents the identity matrix, σ represents the activation function, express The degree matrix, W k Represents the learnable weight matrix set according to the partitioning strategy in step S2, divided by connection type, i.e. l i (v j )=1 represents the physical connection and l i (v j )=2 use two different learnable weight matrices respectively.

[0043] The specific implementation expression of graph convolution is as follows:

[0044]

[0045] w(v i ,v j )=w′.l i (v j ) /

[0046] Among them, f in (v i ,v j ) represents node v i ,v j Input, f out (v i ) represents node v i The spatial features extracted by graph convolution, B(v i ) represents node v i In the corresponding adjacency matrix A k All adjacent point sets in , is a normalization term, and Z i (v i )=|{v k |l i (v k )=l i (v j )}|,v k Indicates that v j Nodes with the same connection type, w ′ () represents a learnable weight matrix, which gives different values ​​according to the input graph node type, and the weight value is trainable; li Indicates that the node v is divided according to the graph partitioning strategy i Labels of adjacent points divided, l of adjacent points of the same type i (v j ) are the same, w(v i ,v j ) indicates that according to v i ,v j The weight value obtained by the type.

[0047] Then add a global information point v in the graph N+1 To achieve global feature information aggregation, this node is connected to all skeleton nodes for graph convolution. The weight matrix of the second type of logical connection is used, which is expressed as follows:

[0048]

[0049] w(v N+1 ,v j )=w′(2)

[0050] After using GCN to achieve information aggregation and spatial feature extraction, the Res residual block is used to prevent the gradient from disappearing. The expression is as follows:

[0051] res(gcn(X,A))=X+gcn(X,A)

[0052] Finally, after using multiple adjacency matrices for graph convolution, the convolution results of each adjacency matrix are linearly combined to obtain the final skeleton key point feature information, completing the spatial feature extraction.

[0053] S42, extracting temporal features through the multi-head self-attention mechanism in the self-attention layer;

[0054] After aggregating the spatial feature information of each frame in step S41, the feature information of each skeleton key point f out (v i ) is input into the self-attention layer to extract temporal features.

[0055] The feature information originally divided by space is converted into time division, and the feature information of the whole body skeleton joint point in the tth frame is set as x t , then the input feature of the m-th layer TC-GCN block is The output features are

[0056] Where t∈(1,2,3…T).

[0057] For the tth frame, first use the three weight matrices W Query , W Key , WValue Multiply them by the input features to get q t ,k t ,v t , the expression is as follows:

[0058]

[0059] Among them, q t represents the query, k t Indicates key, v t Represents a value.

[0060] Calculate all q t ,k t ,v t After that, use any q t With all k t Calculate the vector dot product to get α k,t , input it into the softmax layer in the self-attention layer to obtain an attention weight between 0 and 1 The expression is as follows:

[0061]

[0062] in, Indicates q t The transpose of the vector, d q,t Indicates all q t The matrix composed of all k t The dot product of the matrices composed of .

[0063] Then get and the corresponding v1, v2…v t …v T Multiply to get the same as the input feature The corresponding output features Sum the output features to get the total output feature X of all frames m+1 , which is the temporal feature output in the m-th layer TC-GCN block structure, is expressed as follows:

[0064]

[0065]

[0066] After multi-layer feature extraction, the temporal features output by the final Mth layer are input into the fully connected layer, and step S5 is entered.

[0067] Furthermore, the step S5 is specifically as follows:

[0068] After stacking M layers of TC-GCN blocks, the output features of all N human key points are aggregated through the fully connected layer in the recognition model. A softmax normalization operation is then performed through the softmax layer to convert the output vector into a probability distribution, so that the value of each element in the vector is between 0 and 1, and the sum of all elements is 1. The recognition result is obtained based on the probability distribution, that is, the output of the graph structure skeleton sequence information with a channel number of 25. The 25 channels represent the probability that the input action sequence is one of the 24 Tai Chi moves or not a Tai Chi move.

[0069] The loss function is set to the cross entropy loss function, and the expression is as follows:

[0070]

[0071] Where n represents the number of categories, i.e. 25, Represents a 25-channel vector of global information points, y represents the ground truth category, and T represents the transpose operation.

[0072] Beneficial effects of the present invention: The method of the present invention first uses a human posture estimation algorithm to extract a human skeleton sequence from the collected Tai Chi video, then constructs a graph convolution adjacency matrix based on the prior features of Tai Chi activities, and constructs a Tai Chi action recognition model based on graph convolution and self-attention mechanism, aggregates skeleton data through graph convolution, extracts spatial features, and then extracts temporal features through the multi-head self-attention mechanism in the self-attention layer, and finally completes classification through the fully connected layer, calculates the action classification results, and completes the Tai Chi action classification and recognition. The method of the present invention utilizes graph convolution and self-attention mechanism to realize the recognition and classification of human body movements in Tai Chi videos, uses graph convolution method to calculate the spatial correlation of each joint point, accelerates the aggregation of aggregated information of endpoints with a long physical distance, and uses self-attention mechanism to calculate the temporal correlation of each frame in parallel, thereby enhancing the accuracy and stability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a flowchart of a Tai Chi movement recognition method based on graph convolution and self-attention mechanism of the present invention.

[0074] Figure 2 Schematic diagram of the human skeleton in an embodiment of the present invention.

[0075] Figure 3 This is a structural diagram of the Tai Chi movement recognition model based on graph convolution and self-attention mechanism in an embodiment of the present invention.

[0076] Figure 4 1 is a structural diagram of the m-th layer TC-GCN block in an embodiment of the present invention.

[0077] Figure 5Schematic diagram of the multi-head self-attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The method of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0079] like Figure 1 As shown in the figure, a flow chart of a Tai Chi action recognition method based on graph convolution and self-attention mechanism of the present invention is shown in the figure. The specific steps are as follows:

[0080] S1, collect videos of Tai Chi training process, and use human posture estimation algorithm to extract human skeleton sequence from the collected Tai Chi videos;

[0081] In this embodiment, the human body posture estimation algorithm VideoPose3D is used; the human body skeleton diagram is as follows Figure 2 shown.

[0082] S2. Based on step S1, construct a graph convolution adjacency matrix according to the prior features of Tai Chi activities;

[0083] S3. Build a Tai Chi action recognition model based on graph convolution and self-attention mechanism;

[0084] like Figure 3 As shown, the recognition network model includes: M layers of TC-GCN blocks, fully connected layers, and softmax layers; and the structure of TC-GCN block includes: 1*1 convolution layer, GCN graph convolution layer, self-attention layer, and Res residual block. The structure of the mth layer TC-GCN block is as follows Figure 4 shown.

[0085] S4. Based on the model built in step S3, the skeleton data is aggregated through graph convolution to extract spatial features, and then the temporal features are extracted through the multi-head self-attention mechanism in the self-attention layer;

[0086] Among them, the multi-head self-attention mechanism is as follows Figure 5 shown.

[0087] S5. Based on step S4, complete the classification through the fully connected layer, calculate the action classification result, and complete the Tai Chi action classification and recognition.

[0088] In this embodiment, step S2 is specifically as follows:

[0089] S21, physical prior setting of human skeleton;

[0090] Set the adjacency matrix A according to the physical connection of each key point of the human body 1 , set the i-th key point v i and the jth key point v jIf they are adjacent on the human body, then let A 1 (i,j)=1, otherwise A 1 (i,j)=0.

[0091] If no special graph structure design is made for adjacent nodes, use l i (v j ) represents the connection type classification when performing graph convolution on key points i and j. In this figure, l i (v j )=1,l j (v i )=1, indicating the first type of connection, that is, physical connection.

[0092] S22, prior setting of Tai Chi movement characteristics;

[0093] Since each layer of GCN can only aggregate information from adjacent nodes, the speed of expanding the receptive field is limited, and information loss will occur when two distant nodes interact. Tai Chi is a whole-body exercise that requires coordination of all parts of the body. For example, the coordinated movement of the left and right palms, while the skeleton nodes representing the left and right palms are extremely far apart in the physical graph structure.

[0094] In order to solve the above problems, the adjacency matrix is ​​set according to the characteristics and requirements of Tai Chi exercises.

[0095] (1) Gesture requirements;

[0096] Tai Chi requires coordination of both hands. Set the adjacency matrix A 2 Capturing the Tai Chi hand gestures requires corresponding features. Preserving the original physical connection between the hands and shoulders, the A between the corresponding nodes 2 (v i ,v j )=1, set the physically connected adjacent points to the same subset, l i (v j )=1,l j (v i )=1.

[0097] Establish left-right relative logical connections between the key points corresponding to the left and right palms, arms, and shoulders, and the value A of the adjacency matrix between the corresponding nodes 2 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2, indicating the second type of connection, that is, logical connection;

[0098] Among them, some Tai Chi exercises require the relative position of the palm and shoulder, so a connection is established between the palm and the key point above the shoulder. The value A of the adjacency matrix between the corresponding nodes 2 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2.

[0099] The connections of the remaining graph nodes in the adjacency matrix are set to 0.

[0100] (2) Leg posture requirements;

[0101] Tai Chi requires the legs to be in corresponding postures at different stages of the moves. Set the adjacency matrix A 3 Capturing the leg posture of Tai Chi requires corresponding characteristics. The original physical connection from the toes to the thighs, the A between the corresponding nodes 3 (v i ,v j )=1, set the physically connected adjacent points to the same subset, l i (v j )=1,l j (v i )=1.

[0102] Establish left-right relative logical connections between the key points corresponding to the left and right ankles, knee joints and thigh hip joints, and the A between the corresponding nodes 3 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2.

[0103] The remaining graph structure connections are set to 0.

[0104] (3) Head posture requirements;

[0105] Tai Chi exercises require the athlete to look in the right or left direction. Depending on the stage of the move, the athlete must look towards the palm of his hand or straight ahead. Set the adjacency matrix A 4 Capturing the head posture of Tai Chi requires corresponding features. Establish logical connections between binocular key points and palm key points, and the A between corresponding nodes 4 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (vj )=2,l j (v i )=2.

[0106] The shoulder position is used to determine the front of the human body posture (the vector direction orthogonal to the shoulder line and parallel to the ground), and a left-right relative logical connection is established between the shoulder key points and the binocular key points. The A between the corresponding nodes 4 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2.

[0107] The remaining graph structure connections are set to 0.

[0108] (4) body trunk rotation characteristics;

[0109] In some Tai Chi movements, the athlete frequently rotates the body, such as Cloud Hands. Set the adjacency matrix A 5 Capture the trunk rotation characteristics of Tai Chi. Obtain the trunk rotation characteristics through the relative movement of the center key point of the human pelvis and the shoulders and legs, establish a logical connection between the center key point of the human pelvis and the shoulder key points, and the A between the corresponding nodes 5 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2.

[0110] The physical connection between the central key point of the human pelvis and the hip joints of the legs is retained. 5 (v i ,v j )=1, set the physically connected adjacent points to the same subset, l i (v j )=1,l j (v i )=1.

[0111] The remaining graph structure connections are set to 0.

[0112] In this embodiment, step S4 is specifically as follows:

[0113] S41, aggregate skeleton data through graph convolution and extract spatial features;

[0114] like Figure 3As shown, the skeleton sequence data X obtained in step S1 and the multiple adjacency matrices obtained in steps S21 and S22 (combined with the adjacency matrix set A of Tai Chi movement characteristics) are combined. K ) are respectively input into the M-layer TC-GCNblock network structure of the recognition model described in step S3.

[0115] Among them, the skeleton sequence data of the input m-th layer TC-GCN block is represented as X m [T,N,C m ], N represents the total number of input human skeleton joint points, Figure 3 、 4 [N, N] represents the size of each adjacency matrix dimension, T represents the number of frames in the input skeleton sequence, and the optimal number of layers M of the TC-GCN block structure is determined according to the actual situation (in this embodiment, M = 4 can obtain the best performance), C m Represents the dimension of the data input to the m-th layer TC-GCN block structure, C1 represents the data dimension of the input to the first layer TC-GCN block structure, that is, the original input data. The original input data is 3D skeleton data, then C1 = 3, and the data dimension of the input to the m-th layer TC-GCN block is the data dimension of the output of the m-1-th layer TC-GCN block.

[0116] like Figure 4 As shown, the 1*1 convolution kernel is used in the TC-GCN block network structure to change the number of feature channels and divide the K skeleton posture multi-channel information. And input L layers of GCN to extract spatial information, and use the graph convolution method to aggregate the relevant skeleton node information on a single frame. The expression is as follows:

[0117]

[0118] Among them, l represents the lth layer of GCN, with a total of L layers; X (l) A represents the input of the first layer of the network, A represents the adjacency matrix used for graph convolution calculation, which is used to represent the connection relationship between each graph node, K represents the total number of adjacency matrices used, and K = 5. k represents the kth adjacency matrix (1≤k≤K), each adjacency matrix is ​​set according to step S2, Indicates A k The matrix after adding the self-loop, and I represents the identity matrix, σ represents the activation function, express The degree matrix, W k Represents the learnable weight matrix set according to the graph partitioning strategy in step S2 (i.e., dividing the skeleton connection into physical connection and logical connection), divided by connection type, i.e., l i (vj )=1 represents the physical connection and l i (v j )=2 use two different learnable weight matrices respectively.

[0119] The specific implementation expression of graph convolution is as follows:

[0120]

[0121] w(v i ,v j )=w′.l i (v j ) /

[0122] Among them, f in (v i ,v j ) represents node v i ,v j Input, f out (v i ) represents node v i The spatial features extracted by graph convolution, B(v i ) represents node v i In the corresponding adjacency matrix A k All adjacent point sets in , is a normalization term, and Z i (v i )=|{v k |l i (v k )=l i (v j )}|,v k Indicates that v j Nodes with the same connection type, w ′ () represents a learnable weight matrix, which gives different values ​​according to the input graph node type, and the weight value is trainable; l i Indicates that the node v is divided according to the graph partitioning strategy i Labels of adjacent points divided, l of adjacent points of the same type i (v j ) are the same, w(v i ,v j ) indicates that according to v i ,v j The weight value obtained by the type.

[0123] Then add a global information point v in the graph N+1 To achieve global feature information aggregation, this node is connected to all skeleton nodes for graph convolution. The weight matrix of the second type of logical connection is used, which is expressed as follows:

[0124]

[0125] w(v N+1 ,v j )=w′(2)

[0126] Since global information points and skeleton joint points are not physically connected to each other but have a logical relationship, the weight matrix of the second type of logical connection is used.

[0127] After using GCN to achieve information aggregation and spatial feature extraction, the Res residual block is used to prevent the gradient from disappearing. The expression is as follows:

[0128] res(gcn(X,A))=X+gcn(X,A)

[0129] Finally, after using multiple adjacency matrices for graph convolution, the convolution results of each adjacency matrix are linearly combined to obtain the final skeleton key point feature information, completing the spatial feature extraction.

[0130] S42, extracting temporal features through the multi-head self-attention mechanism in the self-attention layer;

[0131] like Figure 5 As shown, after aggregating the single-frame spatial feature information of each frame in step S41, the feature information of each skeleton key point f out (v i ) is input into the self-attention layer to extract temporal features.

[0132] The feature information originally divided by space is converted into time division, and the feature information of the whole body skeleton joint point in the tth frame is set as x t , then the input feature of the m-th layer TC-GCN block is The output features are

[0133] Where t∈(1,2,3…T).

[0134] For the tth frame, first use the three weight matrices W Query , W Key , W Value Multiply them by the input features to get q t ,k t ,v t , the expression is as follows:

[0135]

[0136]

[0137] Among them, q trepresents query, k t Indicates key (Key), v t Indicates a value.

[0138] Calculate all q t ,k t ,v t After that, use any q t With all k t Calculate the vector dot product to get α k,t , input it into the softmax layer in the self-attention layer to obtain an attention weight between 0 and 1 The expression is as follows:

[0139]

[0140] in, Indicates q t The transpose of the vector, d q,t Indicates all q t The matrix composed of all k t The dot product of the matrices composed of .

[0141] Then get and the corresponding v1, v2…v t …v T Multiply to get the same as the input feature The corresponding output features Sum the output features to get the total output feature X of all frames m+1 , which is the temporal feature output in the m-th layer TC-GCN block structure, is expressed as follows:

[0142]

[0143] After multi-layer feature extraction, the temporal features output by the final Mth layer are input into the fully connected layer, and step S5 is entered.

[0144] In this embodiment, step S5 is specifically as follows:

[0145] After stacking M layers of TC-GCN blocks, the output features of all N human key points are aggregated through the fully connected layer in the recognition model. A softmax normalization operation is then performed through the softmax layer to convert the output vector into a probability distribution, so that the value of each element in the vector is between 0 and 1, and the sum of all elements is 1. The recognition result is obtained based on the probability distribution, that is, the output of the graph structure skeleton sequence information with a channel number of 25. The 25 channels represent the probability that the input action sequence is one of the 24 Tai Chi moves or not a Tai Chi move.

[0146] The loss function is set to the cross entropy loss function, and the expression is as follows:

[0147]

[0148] Where n represents the number of categories, i.e. 25, Represents a 25-channel vector of global information points, y represents the ground truth category, and T represents the transpose operation.

[0149] In summary, the method of the present invention uses an improved graph convolutional network to extract the spatial features of skeleton key points, uses a variety of adjacency matrices that meet the requirements of Tai Chi exercises in addition to physical connections to accelerate the aggregation between key points in the graph structure, and extracts the temporal features between video frames through a multi-head self-attention mechanism. The multi-head self-attention mechanism can perform efficient parallel computing and better identify and extract the feature information of more important frames in the video.

[0150] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A Tai Chi action recognition method based on graph convolution and self-attention mechanism, the specific steps are as follows: S1, collect videos of Tai Chi training process, and use human posture estimation algorithm to extract human skeleton sequence from the collected Tai Chi videos; S2. Based on step S1, construct a graph convolution adjacency matrix according to the prior features of Tai Chi activities; S3. Build a Tai Chi action recognition model based on graph convolution and self-attention mechanism; The recognition network model includes: M layers of TC-GCN blocks, fully connected layers, and softmax layers; and the structure of the TC-GCN block includes: 1*1 convolution layer, GCN graph convolution layer, self-attention layer, and Res residual block; S4. Based on the model built in step S3, the skeleton data is aggregated through graph convolution to extract spatial features, and then the temporal features are extracted through the multi-head self-attention mechanism in the self-attention layer; S5. Based on step S4, complete the classification through the fully connected layer, calculate the action classification result, and complete the Tai Chi action classification and recognition.

2. The Tai Chi action recognition method based on graph convolution and self-attention mechanism according to claim 1 is characterized in that: The step S2 is specifically as follows: S21, physical prior setting of human skeleton; Set the adjacency matrix A according to the physical connection of each key point of the human body 1 , set the i-th key point v i and the jth key point v j If they are adjacent on the human body, then let A 1 (i,j)=1, otherwise A 1 (i,j)=0; If no special graph structure design is made for adjacent nodes, use l i (v j ) represents the connection type classification when performing graph convolution on key points i and j. In this figure, l i (v j )=1,l j (v i )=1, indicating the first type of connection, that is, physical connection; S22, prior setting of Tai Chi movement characteristics; Set up the adjacency matrix according to the characteristics and requirements of Tai Chi exercises; (1) Gesture requirements; Set the adjacency matrix A 2 Capturing the Tai Chi hand gestures requires corresponding features; retaining the original physical connection from the hands to the shoulders, and the A between the corresponding nodes 2 (v i ,v j )=1, set the physically connected adjacent points to the same subset, l i (v j )=1,l j (v i )=1; Establish left-right relative logical connections between the key points corresponding to the left and right palms, arms, and shoulders, and the value A of the adjacency matrix between the corresponding nodes 2 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2, indicating the second type of connection, that is, logical connection; Among them, some Tai Chi techniques require the relative position of the palm and shoulder, so a connection is established between the palm and the key point of the upper shoulder; the value A of the adjacency matrix between the corresponding nodes 2 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2; The connections of the remaining graph nodes in the adjacency matrix are set to 0; (2) Leg posture requirements; Set the adjacency matrix A 3 Capturing the leg posture of Tai Chi requires corresponding characteristics; the original physical connection from the toes to the thighs, the A between the corresponding nodes 3 (v i ,v j )=1, set the physically connected adjacent points to the same subset, l i (v j )=1,l j (v i )=1; Establish left-right relative logical connections between the key points corresponding to the left and right ankles, knee joints and thigh hip joints, and the A between the corresponding nodes 3 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2; The rest of the graph structure connections are set to 0; (3) Head posture requirements; Set the adjacency matrix A 4 Capturing the head posture of Tai Chi requires corresponding features; establishing logical connections between binocular key points and palm key points, and A between corresponding nodes 4 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2; Use the shoulder position to determine the front of the human body posture, establish a left-right relative logical connection between the shoulder key points and the binocular key points, and the A between the corresponding nodes 4 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2; The rest of the graph structure connections are set to 0; (4) body trunk rotation characteristics; Set the adjacency matrix A 5 Capture the trunk rotation characteristics of Tai Chi; obtain the trunk rotation characteristics through the relative movement of the center key point of the human pelvis and the shoulders and legs, establish a logical connection between the center key point of the human pelvis and the shoulder key points, and the A between the corresponding nodes 5 (v i ,v j )=1, the adjacent points of the logical connection are set to the same subset, l i (v j )=2,l j (v i )=2; The central key point of the human pelvis position and the physical connection of the hip joints of the legs are retained; the A between the corresponding nodes 5 (v i ,v j )=1, set the physically connected adjacent points to the same subset, l i (v j )=1,l j (v i )=1; The remaining graph structure connections are set to 0.

3. The Tai Chi action recognition method based on graph convolution and self-attention mechanism according to claim 1 is characterized in that: The step S4 is specifically as follows: S41, aggregate skeleton data through graph convolution and extract spatial features; The skeleton sequence data X obtained in step S1 and the multiple adjacency matrices obtained in steps S21 and S22 are input into the steps In the M-layer TC-GCN block network structure of the recognition model described in S3; Among them, the skeleton sequence data of the input m-th layer TC-GCN block is represented as X m [T,N,C m ], N represents the total number of human skeleton joints input, T represents the number of frames of the input skeleton sequence, and the optimal number of layers M and C of the TC-GCN block structure are determined according to the actual situation. m Represents the dimension of the data input to the m-th layer TC-GCN block structure, C1 represents the data dimension of the input to the first layer TC-GCN block structure, that is, the original input data. The original input data is 3D skeleton data, so C1 = 3, and the data dimension of the input to the m-th layer TC-GCN block is the data dimension of the output of the m-1-th layer TC-GCN block; Then, in the TC-GCN block network structure, a 1*1 convolution kernel is used to change the number of feature channels and divide the K skeleton posture multi-channel information into Input L layers of GCN to extract spatial information and use graph convolution to aggregate relevant skeleton node information on a single frame. The expression is as follows: Among them, l represents the lth layer of GCN, with a total of L layers; X (l) represents the input of the network layer l, A represents the adjacency matrix used for graph convolution calculation, which is used to represent the connection relationship between each graph node, K represents the total number of adjacency matrices used, and K = 5; A k represents the kth adjacency matrix (1≤k≤K), each adjacency matrix is ​​set according to step S2, Indicates A k The matrix after adding the self-loop, and I represents the identity matrix, σ represents the activation function, express The degree matrix, W k Represents the learnable weight matrix set according to the partitioning strategy in step S2, divided by connection type, i.e. l i (v j )=1 represents the physical connection and l i (v j )=2 logical connections use two different learnable weight matrices respectively; The specific implementation expression of graph convolution is as follows: w(v i ,v j )=w′.l i (v j ) / Among them, f in (v i ,v j ) represents node v i ,v j Input, f out (v i ) represents node v i The spatial features extracted by graph convolution, B(v i ) represents node v i In the corresponding adjacency matrix A k All adjacent point sets in , is a normalization term, and Z i (v i )=|{v k |l i (v k )=l i (v j )}|,v k Indicates that v j Nodes with the same connection type, w ′ () represents a learnable weight matrix, which gives different values ​​according to the input graph node type, and the weight value is trainable; l i Indicates that the node v is divided according to the graph partitioning strategy i Labels of adjacent points divided, l of adjacent points of the same type i (v j ) are the same, w(v i ,v j ) indicates that according to v i ,v j The weight value obtained by the type; Then add a global information point v in the graph N+1 To achieve global feature information aggregation, this node is connected to all skeleton nodes for graph convolution. The weight matrix of the second type of logical connection is used, which is expressed as follows: w(v N+1 ,v j )=w′(2) After using GCN to achieve information aggregation and spatial feature extraction, the Res residual block is used to prevent the gradient from disappearing. The expression is as follows: res(gcn(X,A))=X+gcn(X,A) Finally, after using multiple adjacency matrices for graph convolution, the convolution results of each adjacency matrix are linearly combined to obtain the final skeleton key point feature information, completing the spatial feature extraction; S42, extracting temporal features through the multi-head self-attention mechanism in the self-attention layer; After aggregating the spatial feature information of each frame in step S41, the feature information of each skeleton key point f out (v i ) Input into the self-attention layer to extract temporal features; The feature information originally divided by space is converted into time division, and the feature information of the whole body skeleton joint point in the tth frame is set as x t , then the input feature of the m-th layer TC-GCN block is The output features are Where, t∈(1,2,3…T); For the tth frame, first use the three weight matrices W Query , W Key , W Value Multiply them by the input features to get q t ,k t ,v t , the expression is as follows: Among them, q t represents the query, k t Indicates key, v t Represents a value; Calculate all q t ,k t ,v t After that, use any q t With all k t Calculate the vector dot product to get α k,t , input it into the softmax layer in the self-attention layer to obtain an attention weight between 0 and 1 The expression is as follows: in, Indicates q t The transpose of the vector, d q,t Indicates all q t The matrix composed of all k t The dot product of the matrices composed of; Then get and the corresponding v1, v2…v t …v T Multiply to get the same as the input feature The corresponding output features Sum the output features to get the total output feature X of all frames m+1 , which is the temporal feature output in the m-th layer TC-GCN block structure, is expressed as follows: After multi-layer feature extraction, the temporal features output by the final Mth layer are input into the fully connected layer, and step S5 is entered.

4. The Tai Chi action recognition method based on graph convolution and self-attention mechanism according to claim 1, characterized in that: The step S5 is specifically as follows: After stacking M layers of TC-GCN blocks, the output features of all N human key points are aggregated through the fully connected layer in the recognition model. A softmax normalization operation is then performed through the softmax layer to convert the output vector into a probability distribution, so that the value of each element in the vector is between 0 and 1, and the sum of all elements is 1. The recognition result is obtained based on the probability distribution, that is, the output of the graph structure skeleton sequence information with 25 channels. The 25 channels represent the probability that the input action sequence is one of the 24 Tai Chi moves or not a Tai Chi move. The loss function is set to the cross entropy loss function, and the expression is as follows: Where n represents the number of categories, i.e. 25, Represents a 25-channel vector of global information points, y represents the ground truth category, and T represents the transpose operation.

Images