Graph convolution human skeleton action recognition method based on structural attention mechanism

Abstract

The application relates to a graph convolution human skeleton action recognition method based on a structural attention mechanism, and comprises the following steps: S1, constructing a structural attention module, including two submodules of a body part relationship-based attention module and a body symmetry node trajectory-based attention module; S2, adding the body part relationship-based attention module to an adaptive adjacency matrix of an intermediate result of graph convolution to obtain an adaptive adjacency matrix of reinforced body part structure; S3, constructing a related attention module by using the relationship of body symmetry node trajectories to reinforce the attention of a human skeleton action recognition model to body symmetry; and S4, feeding the features reinforced by the structural attention into a classifier to obtain scores corresponding to each action, so that a final recognition result is obtained. Compared with the prior art, the application considers the human structural behavior mechanism in action recognition, and has the advantages of high recognition precision.

Description

Technical Field

[0001] The present invention relates to the field of human skeleton action recognition, and in particular to a graph convolution method for human skeleton action recognition based on a structural attention mechanism. Background Art

[0002] As an extremely important component of computer vision and a highly active research topic, action recognition has garnered increasing attention in recent years. It has found widespread application in fields such as security monitoring, human-computer interaction, and information retrieval. Skeleton-based action recognition methods, in particular, have attracted widespread attention due to their robustness to environmental noise, making them a popular research topic.

[0003] Early deep learning methods used CNNs or RNNs to learn features from skeleton joints. However, these methods are less efficient than methods based on graph convolutional networks (GCNs), because graph convolutions fully exploit the non-Euclidean structure of graph structures and can more naturally handle the laws of human motion. ST-GCN was the first to apply graph convolutions to skeleton action recognition. However, the aggregation of joint features in ST-GCN is designed for a fixed body structure and is not adaptable to flexible motion changes. Summary of the Invention

[0004] This invention is designed to solve the above problems and aims to provide a graph convolution method for human skeleton action recognition based on a structured attention mechanism that can flexibly and efficiently process the laws of human motion. The invention adopts the following technical solutions:

[0005] The present invention provides a graph convolution method for human skeleton action recognition based on a structural attention mechanism, which is characterized by comprising the following steps: step S1, constructing a human skeleton action recognition model based on a structural attention mechanism; step S2, training the human skeleton action recognition model; step S3, inputting a human action picture into the trained human skeleton action recognition model to obtain scores for each predetermined action, thereby obtaining a human skeleton action recognition result; wherein step S1 includes the following sub-steps: step S1-1, constructing a first attention module based on the relationship between body parts, which is used to strengthen the attention paid to different parts of the body and the relationship between different parts, thereby strengthening the attention paid to different parts of the body and the relationship between different parts. The contribution of the more important parts of the body skeleton movement to the recognition of human skeleton movement; step S1-2, adding the first attention module to the intermediate result of the graph convolution, that is, the adaptive adjacency matrix, to obtain the adaptive adjacency matrix that strengthens the structure of the body part; step S1-3, constructing a second attention module based on the symmetrical node trajectory of the body, which is used to explicitly introduce the symmetrical node relationship of the human skeleton, and use the changing trajectory of the symmetrical node relationship during the entire movement process to enhance the accuracy of skeleton movement recognition; in step S3, the features after structural attention enhancement are sent to the classifier of the human skeleton movement recognition model to obtain the human skeleton movement recognition result.

[0006] The graph convolution method for human skeleton action recognition based on the structural attention mechanism provided by the present invention may also have such a technical feature, wherein step S1-2 further includes the following sub-steps: step A1, obtaining an adaptive adjacency matrix A based on a predetermined skeleton network C ; Step A2, according to the adaptive adjacency matrix A C The features represented by the middle nodes and the nodes are converted and merged to generate the attention score matrix A based on the relationship between body parts. PC ; Step A3, the attention score matrix A based on the body part relationship PC Split into the attention score matrix A corresponding to each node JC , and reverse the node order according to the node index; step A4, according to the node-based attention score matrix A JC With the adaptive adjacency matrix A C The adaptive adjacency matrix A with enhanced attention to body part relationships is calculated PAC .

[0007] The graph convolution method for human skeleton action recognition based on the structural attention mechanism provided by the present invention may also have the following technical features, wherein step S1-3 further includes the following sub-steps: step B1, based on the symmetrical node relationship, dividing the left and right parts of the human skeleton features, and splitting the feature X belonging to the left half of the body Land the characteristic X belonging to the right half of the body R Step B2, according to the feature X belonging to the left half of the body L and the feature X belonging to the right half of the body R Calculate the symmetric trajectory attention matrix Re; Step B3, use the symmetric trajectory attention matrix Re to respectively focus on the feature X belonging to the left half of the body L and the feature X belonging to the right half of the body R Perform symmetric trajectory attention enhancement to generate symmetric trajectory attention enhancement feature X′ L and X′ R ; Step B4, the left part feature X′ after the symmetric trajectory attention is strengthened L and right part feature X′ R Merge into a unified skeleton human features X O .

[0008] The graph convolution human skeleton action recognition method based on the structural attention mechanism provided by the present invention can also have such a technical feature, wherein, step S3 mainly converts the action scores of each part into probabilities through the Softmax() function, and the action with the maximum probability is the final recognition result. At the same time, the cross entropy function is used to calculate the loss for back propagation.

[0009] The graph convolution method for human skeleton action recognition based on the structural attention mechanism provided by the present invention may also have such a technical feature, wherein the body part merging rule described in step A2 is: let the adaptive adjacency matrix A C The dimension is (N, N, C), where N represents the number of skeleton nodes and C represents the feature dimension. C The first and second dimensions can be used to obtain the degree of association between a skeleton node and another skeleton node. Let the nodes belonging to the same body part be N jk , where j and k represent the node in the adaptive adjacency matrix A C The index of the corresponding dimension in n represents the number of nodes belonging to the body part, and the final output body part relationship feature P can be described as: The body part relationship attention matrix A described in step A3 PC The generation rule is: A PC =Softmax(P) The adaptive adjacency matrix A with enhanced attention to body part relations described in step A4 PAC The calculation rules are: A PAC =A JC ·A C .

[0010] The graph convolution method for human skeleton action recognition based on the structural attention mechanism provided by the present invention may also have such a technical feature, wherein the calculation method of the symmetric trajectory attention matrix Re in step B2 is as follows: Re=Tanh(W L X L ×W R X R ), where WX represents a linear transformation of the feature X using the parameter matrix W, which can be a fully connected layer or a one-dimensional convolution; X represents a matrix multiplication, which makes the left half feature X L and the right half feature X R Align multiplication on the feature dimension C; Tanh() represents a nonlinear transformation in the form of y = tanh(x).

[0011] The graph convolution method for human skeleton action recognition based on the structural attention mechanism provided by the present invention may also have such a technical feature, wherein the symmetric trajectory attention enhancement feature X′ in step B3 L and X′ R The calculation method is as follows: X′ L =Re×W′ L X L , X′ R =Re T ×W′ R X R ; Where X represents the matrix multiplication, which makes the feature and trajectory attention matrices aligned and multiplied in the time dimension T; Re T Re is transposed so that the feature X of the left half is L It can be aligned with the second dimension T of Re, and the feature X of the right half R It can be aligned with the third dimension T of Re.

[0012] Functions and effects of the invention

[0013] According to the graph convolution human skeleton action recognition method based on the structural attention mechanism of the present invention, relying on the regularity of the human body structure, it not only physically introduces the structural information of each part of the body, but also logically introduces the structural information of the body's symmetrical motion trajectory, thereby conducting a more comprehensive and in-depth excavation of the human body structure. In addition, in the skeleton action recognition model based on the structural attention mechanism that has been trained by the present invention, the structural attention module can achieve end-to-end training, that is, it can obtain the enhanced adaptive adjacency matrix reinforcement features through the body part relationship attention module, and strengthen the structural features of human motion through the symmetrical trajectory attention module, thereby more accurately identifying human actions and more easily distinguishing easily confused action types, reaching the current advanced level. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 1 is a flow chart of a method for human skeleton action recognition based on graph convolution using a structural attention mechanism in an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram of a method framework constructed according to a workflow according to an embodiment of the present invention;

[0016] Figure 3 1 is a schematic diagram of a baseline network flow chart with a body part relationship attention mechanism added to an embodiment of the present invention;

[0017] Figure 4 This is an example diagram of the corresponding positions of human skeleton nodes in the NTU-RGB+D dataset according to an embodiment of the present invention;

[0018] Figure 5 is a schematic diagram of a flow chart of a symmetric trajectory attention mechanism according to an embodiment of the present invention;

[0019] Figure 6 is a visualization diagram of attention scores of body part relationships during a sitting action according to an embodiment of the present invention;

[0020] Figure 7 is a visualization diagram of the attention weights of the symmetrical trajectories in the sitting action according to an embodiment of the present invention;

[0021] Figure 8 This is a histogram showing the improvement effect of the two attention mechanisms in different action categories in the NW-UCLA dataset of the embodiment of the present invention;

[0022] Figure 9 This is a schematic diagram of the Top-1 accuracy of an embodiment of the present invention on the NW-UCLA validation set;

[0023] Figure 10 This is a schematic diagram of the accuracy of the embodiment of the present invention on the NTU-RGB+D test set;

[0024] Figure 11 FIG4 is a schematic diagram of the accuracy of an embodiment of the present invention on the NTU-RGB+D 120 test set. DETAILED DESCRIPTION

[0025] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following is a detailed description of the graph convolution human skeleton action recognition method based on the structural attention mechanism of the present invention in combination with the embodiments and drawings.

[0026] <Example>

[0027] Figure 1 1 is a flow chart of a method for human skeleton action recognition based on graph convolution using a structural attention mechanism in an embodiment of the present invention; Figure 2 It is a schematic diagram of a method framework constructed according to a workflow according to an embodiment of the present invention.

[0028] like Figure 1 and Figure 2 As shown in the figure, the graph convolution human skeleton action recognition method based on the structural attention mechanism includes the following steps:

[0029] Step S1: constructing a human skeleton action recognition model based on a structural attention mechanism; in this embodiment, step S1 also includes the following sub-steps:

[0030] Step S1-1, constructing a first attention module based on body part relationships, which is used to distinguish the attention of different body parts and the relationships between different parts, and to enhance the contribution of the parts with high importance in human skeletal motion recognition;

[0031] Step S1-2, adding the first attention module to the intermediate result of the graph convolution, that is, the adaptive adjacency matrix, to obtain the adaptive adjacency matrix that enhances the body part structure;

[0032] Figure 3 1 is a schematic diagram of a baseline network flow chart with a body part relationship attention mechanism added to an embodiment of the present invention;

[0033] Wherein, step S1-2 further includes the following sub-steps:

[0034] Step A1: Obtaining an adaptive adjacency matrix A based on a predetermined skeleton network C ;

[0035] Step A2: According to the adaptive adjacency matrix A C The features represented by the middle nodes and the nodes are converted and merged to generate the attention score matrix A based on the relationship between body parts. PC ;

[0036] Specifically, taking the NTU-RGB+D dataset as an example, the conversion rules of body parts from single skeleton nodes to local body parts are defined as follows: the neck, head, left shoulder, right shoulder, and spine nodes are converted to the torso part; the left elbow, left wrist, left hand, left fingertip, and left thumb nodes are converted to the left arm part; the right elbow, right wrist, right hand, right fingertip, and right thumb nodes are converted to the right arm part; the spine base, left hip, left knee, left ankle, and left foot nodes are converted to the left leg part; the spine middle, right hip, right knee, right ankle, and right foot nodes are converted to the right leg part, as shown in the figure. Figure 4 As shown. The body part merging rule is: Let the adaptive adjacency matrix A in the graph convolution be C The dimension is (N,N,C), where N represents the number of skeleton nodes and C represents the feature dimension. Take the adaptive adjacency matrix A CThe first and second dimensions can be used to obtain the degree of correlation between a skeleton node and another skeleton node. Let the nodes belonging to the same body part be N jk , where j and k represent the node in the adaptive adjacency matrix A C The index of the corresponding dimension in n represents the number of nodes belonging to the body part, and the final output body part relationship feature P can be described as:

[0037]

[0038] The generation rule of the body part attention matrix is: A PC =Softmax(P).

[0039] Step A3: transform the attention score matrix A based on the body part relationship into PC Split into the attention score matrix A corresponding to each node JC , and reverse the node order according to the node index;

[0040] Step A4: According to the node-based attention score matrix A JC With the adaptive adjacency matrix A C The adaptive adjacency matrix A with enhanced attention to body part relationships is calculated PAC .

[0041] Specifically, the splitting rule of the body part attention matrix is: assign nodes belonging to the same body part to A PC The feature values ​​with the same attention in the body part relationship are obtained to obtain the attention score matrix A corresponding to each node JC , and reversely transform according to the above transformation rules so that the skeleton nodes return to their original positions according to the index; the adaptive adjacency matrix A with enhanced attention to body part relationships PAC The calculation rules are: A PAC =A JC ·A C .

[0042] Step S1-3, constructing a second attention module based on the symmetric node trajectory of the body, for explicitly introducing the symmetric node relationship of the human skeleton, and utilizing the changing trajectory of the symmetric node relationship throughout the action process to enhance the accuracy of skeleton action recognition;

[0043] Figure 5 is a schematic diagram of a flow chart of a symmetric trajectory attention mechanism according to an embodiment of the present invention;

[0044] In this embodiment, step S1-3 further includes the following sub-steps:

[0045] Step B1: Based on the symmetrical node relationship, the left and right parts of the human skeleton features are divided to separate the features X belonging to the left half of the body. L and the characteristic X belonging to the right half of the body R ;

[0046] Since the human skeleton is split, the nodes in the middle part of the split do not belong to the left half or the right half, so they are not processed;

[0047] Specifically, taking the NTU-RGB+D dataset as an example, the left and right part division rule is defined as: the left shoulder, left elbow, left wrist, left hand, left fingertip, left thumb, left hip, left knee, left ankle, and left foot are divided into the left half feature X L ; Right shoulder, right elbow, right wrist, right hand, right fingertips, right thumb, right hip, right knee, right ankle, right foot are divided into the right half feature X R ; The base of the spine, the middle of the spine, the spine, the neck, and the head are divided into the middle part features, such as Figure 4 shown.

[0048] Step B2, based on the feature X belonging to the left half of the body L and the feature X belonging to the right half of the body R The symmetric trajectory attention matrix Re is calculated;

[0049] Specifically, the symmetric trajectory attention matrix Re is calculated by the following formula:

[0050] Re=Tanh(W L X L ×W R X R ),

[0051] Where WX represents the linear transformation of feature X using parameter matrix W, which can be in the form of a fully connected layer or a one-dimensional convolution; X represents matrix multiplication, which makes the left half of feature X L and the right half feature X R Align and multiply on the feature dimension C; Tanh() represents a nonlinear transformation in the form of y = tanh(x). In this case, the dimensions of Re are (V, T, T), where V represents the number of bilaterally symmetric node pairs and T represents the time dimension.

[0052] Step B3, using the symmetric trajectory attention matrix Re, the feature X belonging to the left half of the body is respectively L and the feature X belonging to the right half of the body R Perform symmetric trajectory attention enhancement to generate symmetric trajectory attention enhancement feature X′ L and X′ R ;

[0053] Specifically, the generated symmetric trajectory attention-enhancing feature X′ L and X′ R The calculation is as follows:

[0054] X′ L =Re×W′ L X L , X′ R =Re T ×W′ R X R

[0055] Among them, the form of WX represents the linear transformation of feature X using the parameter matrix W, which can be in the form of a fully connected layer or a one-dimensional convolution; X represents matrix multiplication, which makes the feature and trajectory attention matrix aligned and multiplied in the time dimension T; Re T Re is transposed so that the feature X in the left half is L It can be aligned with the second dimension T of Re, and the feature X in the right half R It can be aligned with the third dimension T of Re.

[0056] Step B4: Focus the left feature X′ after the symmetric trajectory is strengthened L and right part feature X′ R Merge into a unified skeleton human features X O ;

[0057] Specifically, according to the index of the skeleton node feature, the feature X′ of the left half is L and the feature X′ of the right half R When merging, the features of the two parts do not overlap in index, so the merging rule will not cause ambiguity. The features of the middle nodes are preserved and do not participate in the feature enhancement of the symmetric trajectory attention mechanism. Therefore, the features of the middle nodes do not change during input and output.

[0058] Step S2, training the human skeleton action recognition model;

[0059] Step S3, inputting the human action picture into the trained human skeleton action recognition model to obtain scores for each predetermined action, thereby obtaining a human skeleton action recognition result;

[0060] In this embodiment, in step S3, the features enhanced by structural attention are fed into the classifier of the human skeleton action recognition model to obtain the human skeleton action recognition result. Specifically, the Softmax() function is used to convert the obtained scores of each action into probabilities, and the action with the highest probability is the final recognition result. At the same time, the cross entropy function is used to calculate the loss for backpropagation.

[0061] In order to verify the actual performance of the present invention in the skeleton action recognition task, this embodiment conducted experiments on three well-known public datasets: NW-UCLA, NTU-RGB+D, and NTU-RGB+D120. Since all three datasets contain skeleton action data, which is consistent with most methods, this embodiment uses accuracy as the evaluation criterion. The results are shown in Figure 2. Figures 9-11 shown.

[0062] In order to clarify the effect of structural attention enhancement features on this dataset, Figure 6 、 Figure 7 、 Figure 8 Visualizations are performed, listing the visualization of the attention scores of body part relationships in the sit-down action, the visualization of the attention weights of symmetric trajectories, and the bar charts of the improvement effects of the two attention mechanisms in different action categories. Figure 6 The results show that in terms of attention to body part relationships, the two legs and their relationships with other parts carry more importance tendencies. Figure 7 The results show that the nodes of the two legs generate higher weights than other nodes in the symmetric trajectory attention. This phenomenon is consistent with the part that should be paid attention to in the sitting action. Figure 8 The results show that the action types of the two attention tendencies are different, and to a certain extent, they complement each other, so that working together can achieve better structural attention effects. The above results show that: structural information can effectively assist in identifying action information; the learning of structural attention weights activates the attention module containing the corresponding structural information. This process is similar to how people recognize actions with the naked eye, which shows that this method can help computers further understand human actions. Therefore, for some scenarios where only skeleton nodes are used for action recognition, the present invention can achieve effective utilization of structural information and achieve better results.

[0063] Example Function and Effect

[0064] According to the graph convolution human skeleton action recognition method based on the structural attention mechanism of the present invention, relying on the regularity of the human body structure, it not only physically introduces the structural information of each part of the body, but also logically introduces the structural information of the body's symmetrical motion trajectory, thereby conducting a more comprehensive and in-depth excavation of the human body structure. In addition, in the skeleton action recognition model based on the structural attention mechanism that has been trained by the present invention, the structural attention module can achieve end-to-end training, that is, it can obtain the enhanced adaptive adjacency matrix reinforcement features through the body part relationship attention module, and strengthen the structural features of human motion through the symmetrical trajectory attention module, thereby more accurately identifying human actions and more easily distinguishing easily confused action types, reaching the current advanced level.

[0065] The above experimental results demonstrate that structural information can effectively assist in identifying action information; learning structural attention weights activates attention modules that incorporate the corresponding structural information. This process is similar to how humans recognize actions with the naked eye, demonstrating that this method can help computers further understand human actions. Therefore, for scenarios where action recognition relies solely on skeleton nodes, this method can effectively utilize structural information and achieve superior results.

[0066] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A graph convolution method for human skeleton action recognition based on structural attention mechanism, characterized by: The following steps are involved: Step S1, constructing a human skeleton action recognition model based on a structural attention mechanism; Step S2, training the human skeleton action recognition model; Step S3, inputting the human action picture into the trained human skeleton action recognition model to obtain scores for each predetermined action, thereby obtaining a human skeleton action recognition result; Wherein, step S1 includes the following sub-steps: Step S1-1, constructing a first attention module based on body part relationships, which is used to distinguish the attention of different body parts and the relationships between different parts, and to enhance the contribution of the parts with high importance in human skeletal motion recognition; Step S1-2, adding the first attention module to the intermediate result of the graph convolution, that is, the adaptive adjacency matrix, to obtain the adaptive adjacency matrix that enhances the body part structure; Step S1-3, constructing a second attention module based on the symmetric node trajectory of the body, for explicitly introducing the symmetric node relationship of the human skeleton, and utilizing the changing trajectory of the symmetric node relationship throughout the action process to enhance the accuracy of skeleton action recognition; In step S3, the features after structural attention enhancement are fed into the classifier of the human skeleton action recognition model to obtain the human skeleton action recognition result. Step S1-3 includes the following sub-steps: Step B1: Based on the symmetrical node relationship, the left and right parts of the human skeleton features are divided to separate the features X belonging to the left half of the body. L and the characteristic X belonging to the right half of the body R ; Step B2, based on the feature X belonging to the left half of the body L and the feature X belonging to the right half of the body R The symmetric trajectory attention matrix Re is calculated; Step B3, using the symmetric trajectory attention matrix Re, the feature X belonging to the left half of the body is respectively L and the feature X belonging to the right half of the body R Perform symmetric trajectory attention enhancement to generate the feature X′ of the left half after symmetric trajectory attention enhancement L and the feature X′ of the right half R ; Step B4: Focus the left half of the symmetric trajectory on the feature X′ L and the feature X′ of the right half R Merge into a unified skeleton human features X O .

2. The graph convolution method for human skeleton action recognition based on the structured attention mechanism according to claim 1, Its characteristics are: Wherein, step S1-2 further includes the following sub-steps: Step A1: Obtaining an adaptive adjacency matrix A based on a predetermined skeleton network C ; Step A2: According to the adaptive adjacency matrix A C The nodes in the body are transformed and merged with the features represented between the nodes to generate the attention score matrix A based on the relationship between body parts. PC ; Step A3: transform the attention score matrix A based on the body part relationship into PC Split into the attention score matrix A corresponding to each node JC , and reverse the node order according to the node index; Step A4: According to the node-based attention score matrix A JC With the adaptive adjacency matrix A C The adaptive adjacency matrix A with enhanced attention to body part relationships is calculated PAC .

3. The graph convolution-based human skeleton action recognition method based on structured attention mechanism according to claim 1, characterized in that: in, In step S3, the scores of each action are converted into probabilities through the Softmax() function. The action with the highest probability is the final recognition result. At the same time, the cross entropy function is used to calculate the loss for back propagation.

4. The method for human skeleton action recognition based on graph convolution and structured attention mechanism according to claim 2, characterized in that: in, The body part merging rule in step A2 is: let the adaptive adjacency matrix A C The dimension is (N, N, C), where N represents the number of skeleton nodes and C represents the feature dimension. Take the adaptive adjacency matrix A C The first and second dimensions can be used to obtain the degree of association between a skeleton node and another skeleton node. Let the nodes belonging to the same body part be N jk , where j and k represent the node in the adaptive adjacency matrix A C The index of the corresponding dimension in , n represents the number of nodes belonging to the body part, and the final output body part relationship feature P can be described as: The body part relationship attention matrix A described in step A2 PC The generation rules are: A PC =Softmax(P); The adaptive adjacency matrix A with enhanced attention to body part relationships described in step A4 PAC The calculation rules are: A PAC =A JC ·A C 。 5. The graph convolution method for human skeleton action recognition based on structured attention mechanism according to claim 1, characterized in that: in, The calculation method of the symmetric trajectory attention matrix Re in step B2 is as follows: Re=Tanh(W L X L ×W R X R ) Where WX represents the linear transformation of feature X using parameter matrix W, which can be in the form of a fully connected layer or a one-dimensional convolution; X represents matrix multiplication, which makes the left half feature X L and the right half feature X R Align multiplication on the feature dimension C; Tanh() represents a nonlinear transformation in the form of y = tanh(x).

6. The graph convolution method for human skeleton action recognition based on structured attention mechanism according to claim 1, characterized in that: in, The symmetric trajectory attention enhancement feature X′ described in step B3 L and X′ R The calculation method is as follows: X L ′ =Re×W L ′ X L X′ R =Re T ×W R ′ X R Where X represents the matrix multiplication, which makes the feature and trajectory attention matrices aligned and multiplied in the time dimension T; Re T Re is transposed so that the feature X of the left half is L It can be aligned with the second dimension T of Re, and the feature X of the right half R It can be aligned with the third dimension T of Re.

Images