Human skeleton behavior recognition method based on adaptive graph convolution and self-attention

Abstract

The invention provides a human skeleton behavior recognition method based on adaptive graph convolution and self-attention, and relates to the technical field of behavior recognition, and the method comprises the following steps: processing an NTU RGB + D sequence containing K types of human skeleton behaviors; constructing a spatial relationship modeling module; constructing a multi-scale time convolution module; and constructing a hybrid network. According to the technical scheme, the problems that in the prior art, an identification model is poor in robustness to complex actions and non-standard attitudes, and the identification precision is low due to insufficient spatial-temporal feature extraction are solved.

Description

Technical Field

[0001] This invention relates to the technical field of behavior recognition, and more particularly to a method for human skeleton behavior recognition based on adaptive graph convolution and self-attention. Background Technology

[0002] In the field of action recognition in computer vision, skeleton data has become a mainstream research direction due to its strong anti-interference ability and ability to accurately depict the essence of actions. Early methods that relied on manually designed features had weak generalization ability. After the rise of deep learning, solutions based on graph convolutional networks (GCNs) stood out due to their natural adaptability to skeleton topology. Spatiotemporal graph convolutional networks (ST-GCNs), as a milestone model, significantly improved recognition accuracy through a two-stage framework of "spatial convolution to extract joint spatial relationships + temporal convolution to capture dynamic evolution". It is still the basic architecture of mainstream algorithms. Its core is to model human joints as nodes and physical connections as edges in a graph structure. However, the fixed adjacency matrix design of traditional ST-GCN has significant limitations: this matrix is ​​constructed based on the physical connections of the human skeleton and remains unchanged throughout, making it unable to adapt to the dynamic dependency patterns of different movements. For example, the strong connection between the wrist and shoulder during a wave is masked by the physical connection weights. Furthermore, it lacks a self-attention mechanism, making it difficult to capture long-range connections between joints without direct physical connections, resulting in insufficient feature extraction for complex whole-body movements. The fixed topology also cannot adapt to individual differences and posture deformations, easily introducing redundant features. To address these issues, existing research focuses on integrating dynamic adjacency matrix construction with attention mechanisms, but still faces challenges such as high computational complexity and difficulty in balancing global and local features, thus clarifying the key areas for future algorithm improvements. Summary of the Invention

[0003] In view of the technical problems mentioned in the background, a method for human skeleton behavior recognition based on adaptive graph convolution and self-attention is provided.

[0004] The technical means employed in this invention are as follows: A method for human skeleton behavior recognition based on adaptive graph convolution and self-attention includes the following steps: Step 1: Process NTU RGB+D sequences containing K-type human skeletal behaviors. Process it; Step 2: Construct a spatial relationship modeling module; the spatial relationship modeling module includes: a hybrid module AAGC combining adaptive graph convolution and self-attention; the hybrid module AAGC includes: an Adaptive module and a Self-Attention module, the Adaptive module is used to model the local dependencies of key points; the Self-Attention module is used to model the global dependencies of key points; Step 3: Construct a multi-scale temporal convolution module; Step 4: Construct a hybrid network; the hybrid network includes: an Adaptive module, a Self-Attention module, and an MS-TCN.

[0005] Further, step 1 includes the following steps: Step 11: Expand the corresponding features of different key points of M actors in N skeleton sequences at all times, using the following formula: ; in, This represents the input sequence of human skeletal behaviors. This indicates that the shape of the input skeleton behavior sequence is adjusted; Indicates the number of joints in the human skeleton; Indicates the feature dimension of the input; Indicates the number of time frames in the action sequence; Step 12: Normalize the features of all key points of M actors in N action sequences at all times, using the following formula: ; in, This indicates a batch normalization operation; Step 13: For a behavior sequence with two executing entities, split one frame into two frames, and integrate the split results into a single behavior sequence. The formula is as follows: .

[0006] Furthermore, step 2 includes the following steps: Step 21: The adaptive graph convolution module uses three adjacency matrices to reflect the relationship between different joints in the human skeleton, namely Ak, Bk, and Ck. Where Ak represents the original normalized value. Adjacency matrix; Bk represents Adjacency matrix; Ck represents the data correlation graph; To determine whether a connection exists between two vertices and the strength of that connection, a normalized embedded Gaussian function is used to calculate the similarity between the two vertices, as shown in the formula: ; in, Represents the total number of vertices; Represents vertices ; Represents vertices ; All indicate The embedding function for convolution operations; T represents the transpose operation, and e represents the natural constant. Represents vertices Features Transformed embedding vector; Represents vertices Features Transformed embedding vector; This indicates that for all vertices Summing the numerators; Step 22: Use the dot product to measure the similarity between two vertices in the embedding space.

[0007] Furthermore, step 22 includes the following steps: Step 221: Given the input feature map The size is By embedding functions and embedded functions Embedded into In the middle, the two embedded feature maps are rearranged and reshaped into Matrix and matrix; Step 222, Matrix and Matrix multiplication yields Similarity matrix Simultaneously, the similarity matrix The value is normalized to a value between 0 and 1 and used as a soft edge between two vertices: ; in, and These are the embedded functions and The parameter is initialized to 0.

[0008] Step 223: Add the obtained Ck, Bk, and Ak together, and calculate the output of the adaptive convolutional layer using a weighted function. The formula is as follows: ; in, This represents the weighting function.

[0009] Furthermore, the self-attention module generates query, key, and value vectors, calculates the attention weights between global nodes, and finally aggregates global features.

[0010] Furthermore, the multi-scale temporal convolution module in step 3 includes four parallel branches. The four parallel branches include: a first branch and a second branch; the first branch and the second branch are two temporal convolutions with different receptive fields, as shown in the formula: ; in, Indicates branch The output after temporal convolution; Indicates branch 1×1 convolution operation; This indicates a temporal convolution operation. Indicates the first The size of each branch convolution kernel; Indicates the first The hole rate of a time convolution; This indicates the output of the space module, i.e., the output of the AAGC module; The four parallel branches also include a third branch; the third branch consists of a 1×1 convolution and a max pooling operation, with the following formula: ; in, This represents the output of the max-pooling branch; This represents the max pooling operation; The four parallel branches also include a fourth branch; the fourth branch is a 1×1 convolution, with the formula: ; in, This represents the output of a 1×1 branch; The output of the MS-TCN is obtained by concatenating the outputs of all branches along the channel dimension, as shown in the formula: ; in, This represents the output of the multi-scale temporal convolution module. This indicates a channel splicing operation.

[0011] Furthermore, step 4 includes the following steps: Step 41: For the input T-frame human skeleton sequence, feed it into the Adaptive module and the Self-Attention module respectively to extract spatial features with local and global receptive fields, which are obtained by combining adaptive graph convolution and self-attention. ; Step 42: Analyze the results of spatial modeling. The input is a temporal convolution module used to model the temporal correlation between different frames; Step 43: After L layers of continuous alternating spatial and temporal modeling, the final output features will pass through a global average pooling layer and a fully connected layer in sequence to obtain the behavior classification score.

[0012] Compared with the prior art, the present invention has the following advantages: 1. The dual-branch skeleton feature extraction network that integrates adaptive graph convolution and self-attention provided by this invention combines the dynamic learning ability of adaptive graph convolution on local skeleton topology with the modeling ability of self-attention mechanism on global joint dependencies, thereby achieving refined capture of spatiotemporal features of the human skeleton. It retains the dynamic adaptability of local skeleton connections and also explores the long-distance dependencies between global joints.

[0013] 2. The multi-scale temporal convolution module (MS-TCN) designed in this invention combines temporal convolution kernels with different dilation rates and max pooling branches, which overcomes the limitation of traditional fixed kernel temporal convolution in capturing multi-scale temporal features. It can adaptively extract action features under different time dimensions, thereby improving the recognition accuracy of complex and multi-stage human behavior.

[0014] 3. This invention introduces a learnable adjacency matrix offset parameter into the adaptive graph convolution branch, and constructs a dynamic adjacency matrix through an embedding function and Softmax normalization. This solves the problem that traditional GCNs rely on fixed bone topology and cannot adapt to changes in joint connection relationships under different action postures, and enhances the robustness of the model to non-standard postures and action deformations.

[0015] In summary, by applying the technical solution of this invention, and addressing the shortcomings of existing human skeleton behavior recognition models, such as "inflexible feature extraction due to fixed local skeleton topology, lack of global joint dependency modeling, and insufficient ability to capture multi-scale temporal features," this invention achieves comprehensive and refined extraction of skeleton features through innovative approaches including adaptive graph convolution to dynamically adjust skeleton adjacency relationships, self-attention modeling of global dependencies, and multi-scale temporal convolution to mine temporal features. Therefore, the technical solution of this invention solves the problems of poor robustness to complex actions and non-standard poses, and low recognition accuracy due to insufficient spatiotemporal feature extraction in existing recognition models.

[0016] Based on the above reasons, this invention can be widely applied in fields such as intelligent video surveillance, human-computer interaction, sports motion analysis, rehabilitation exercise assessment, and intelligent security. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the human skeleton behavior recognition method based on self-attention and deformable graph convolution of the present invention.

[0019] Figure 2 This invention is a network module composed of AAGC and MS-TCN.

[0020] Figure 3 This is a structural diagram of the Adaptive module of the present invention.

[0021] Figure 4 This is a structural diagram of the Self-Attention module of the present invention.

[0022] Figure 5 This is a structural diagram of the MS-TCN module of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] like Figure 1-5 As shown, this invention provides a method for human skeleton behavior recognition based on adaptive graph convolution and self-attention, including the following steps: Step 1: Process NTU RGB+D sequences containing K-type human skeletal behaviors. Processing is performed; step 1 includes the following steps: Step 11: Expand the corresponding features of different key points of M actors in N skeleton sequences at all times, using the following formula: ; in, This represents the input sequence of human skeletal behaviors. This indicates that the shape of the input skeleton behavior sequence is adjusted; Indicates the number of joints in the human skeleton; Indicates the feature dimension of the input; Indicates the number of time frames in the action sequence; Step 12: Normalize the features of all key points of M actors in N action sequences at all times, using the following formula: ; in, This indicates a batch normalization operation; Step 13: For a behavior sequence with two executing entities, split one frame into two frames, and integrate the split results into a single behavior sequence. The formula is as follows: .

[0026] Step 2: Construct a spatial relationship modeling module; the spatial relationship modeling module includes: a hybrid module AAGC combining adaptive graph convolution and self-attention; the hybrid module AAGC includes: an Adaptive module and a Self-Attention module, the Adaptive module is used to model the local dependencies of the joints; the Self-Attention module is used to model the global dependencies of the joints.

[0027] Preferably, in this application, step 2 includes the following steps: Step 21: The adaptive graph convolution module uses three adjacency matrices to reflect the relationship between different joints in the human skeleton, namely Ak, Bk, and Ck. Where Ak represents the original normalized value. Adjacency matrix; Bk represents The adjacency matrix contains values ​​indicating whether a connection exists between two joints and the strength of that connection. Bk is initialized to 0, and the values ​​in the matrix can be learned and optimized based on data during training. Ck represents the data correlation graph. To determine whether a connection exists between two vertices and the strength of that connection, a normalized embedded Gaussian function is used to calculate the similarity between the two vertices, as shown in the formula: ; in, Represents the total number of vertices; Represents vertices ; Represents vertices ; All indicate The embedding function for convolution operations; T represents the transpose operation, and e represents the natural constant. Represents vertices Features Transformed embedding vector; Represents vertices Features Transformed embedding vector; This indicates that for all vertices Summing the numerators; Step 22: Use the dot product to measure the similarity between two vertices in the embedding space; Step 221: Given the input feature map The size is By embedding functions and embedded functions Embedded into In the middle, the two embedded feature maps are rearranged and reshaped into Matrix and matrix; Step 222, Matrix and Matrix multiplication yields Similarity matrix Simultaneously, the similarity matrix The value is normalized to a value between 0 and 1 and used as a soft edge between two vertices: ; in, and These are the embedded functions and The parameter is initialized to 0.

[0028] Step 223: Add the obtained Ck, Bk, and Ak together, and calculate the output of the adaptive convolutional layer using a weighted function. The formula is as follows: ; in, This represents the weighting function.

[0029] The self-attention module generates query, key, and value vectors, calculates attention weights between global nodes, and ultimately aggregates global features. Firstly, through... Convolution takes the input feature x (with dimension 1) as an input feature x. Projecting the data onto a unified attention dimension D (the same as the output channel dimension (out_channels) of the adaptive graph convolution module) yields three vectors: Query, Key, and Value, as shown in the formula: ; ; ; in These are the weights of the query, key, and value convolutional layers, respectively, with the output dimension being...

[0030] Next, the mean value is calculated for the time dimension T to compress temporal information, retaining only spatial node features. The formula is: ; ; ; Output dimensions all from Become .

[0031] Then, the similarity between nodes is calculated by the dot product of the query and the key. After scaling (to avoid gradient vanishing) and Softmax (to normalize weights), the final attention weights are obtained, and dropout regularization is applied to prevent overfitting. The formula is as follows: ; in It is the dimension transpose of the query (from) Become The output similarity matrix has dimensions of ), , of which To alleviate The problem of excessively large dot product values ​​and gradient vanishing after softmax is that the final output dimension remains the same. .

[0032] Then use the normalized attention weights The value vector is weighted and summed to aggregate the feature information of all nodes. The formula is as follows: ; in Dimensions , Dimensions The output dimension after matrix multiplication is , Indicates the first Features of each node after fusing global information.

[0033] Since the output dimension of the adaptive graph convolution module is It is necessary to restore the attention features to the time dimension and through Convolution matching output channels, the formula is:

[0034] in It is a dimension of A vector of all 1s, which is transformed through repeated operations. Expand to Aligned with the time dimension of the adaptive graph convolution module. These are the weights of the projected convolutional layer, which change the number of channels from... Adjusted to The final output is attention features. , dimension .

[0035] Finally, attention features Branch features with adaptive graph convolution module By using learnable weights for weighted fusion, the contributions of both are balanced, as shown in the formula: ; in These are learnable parameters (initialized to 0.5), which adaptively adjust the weights of local features and self-attention global features in the adaptive graph convolution module through backpropagation.

[0036] Step 3: Construct a multi-scale temporal convolution module; the multi-scale temporal convolution module in Step 3 includes: 4 parallel branches; wherein, the 4 parallel branches include: a first branch and a second branch; the first branch and the second branch are two temporal convolutions with different receptive fields, the formula is: ; in, Indicates branch The output after temporal convolution; Indicates branch 1×1 convolution operation; This indicates a temporal convolution operation. Indicates the first The size of each branch convolution kernel; Indicates the first The hole rate of a time convolution; This indicates the output of the space module, i.e., the output of the AAGC module; The four parallel branches also include a third branch; the third branch consists of a 1×1 convolution and a max pooling operation, with the following formula: ; in, This represents the output of the max-pooling branch; This represents the max pooling operation; The four parallel branches also include a fourth branch; the fourth branch is a 1×1 convolution, with the formula: ; in, This represents the output of a 1×1 branch; The output of the MS-TCN is obtained by concatenating the outputs of all branches along the channel dimension, as shown in the formula: ; in, This represents the output of the multi-scale temporal convolution module. This indicates a channel splicing operation.

[0037] Step 4: Construct a hybrid network; the hybrid network includes: an Adaptive module, a Self-Attention module, and an MS-TCN. Step 4 includes the following steps: Step 41: For the input T-frame human skeleton sequence, feed it into the Adaptive module and the Self-Attention module respectively to extract spatial features with local and global receptive fields, which are obtained by combining adaptive graph convolution and self-attention. ; Step 42: Analyze the results of spatial modeling. The input is a temporal convolution module used to model the temporal correlation between different frames; Step 43: After L layers of continuous alternating spatial and temporal modeling, the final output features will pass through a global average pooling layer and a fully connected layer in sequence to obtain the behavior classification score.

[0038] Example 1 To verify the effectiveness of the method of this invention, experiments were conducted on the commonly used human skeleton behavior dataset NTU RGB+D 60. To ensure a fair comparison with other methods, the widely used Cross-Subject and Cross-View metrics were used as evaluation indicators, and the Top-1 accuracy was reported.

[0039] Table 1. Action recognition accuracy compared to existing methods on the NTU RGB+D 60 dataset.

[0040] As can be seen from Table 1, the method proposed in this invention achieves top-1 accuracy of 91.3% and 96.7% on the X-sub and X-view evaluation criteria, respectively.

[0041] The experimental results in Table 1 show that the method of the present invention achieves better results on the X-Sub standard compared with other methods, and achieves very competitive results with the latest methods on the X-View standard, proving the effectiveness and feasibility of the method proposed in this invention.

[0042] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human pose skeleton behavior recognition method based on adaptive graph convolution and self-attention, characterized in that, The method comprises the following steps: Step 1, processing NTU RGB+D sequences containing K-class human skeleton behaviors ; Step 2, constructing a spatial relationship modeling module; The spatial relationship modeling module comprises a hybrid module AAGC combining adaptive graph convolution and self-attention; the hybrid module AAGC comprises an Adaptive module and a Self-Attention module, the Adaptive module is used to model the local dependency relationship of the node, and the Self-Attention module is used to model the global dependency relationship of the node; Step 3, constructing a multi-scale time convolution module; Step 4, constructing a hybrid network; the hybrid network comprises an Adaptive module, a Self-Attention module and an MS-TCN.

2. The human pose skeleton behavior recognition method based on adaptive graph convolution and self-attention according to claim 1, characterized in that, The step 1 comprises the following steps: Step 11, the corresponding features of different nodes of M behavior subjects in N skeleton sequences at all times are unfolded, and the formula is: ； wherein, represents an input human pose sequence; represents adjusting the shape of the input pose sequence; represents the number of human pose joints; represents the input feature dimension; represents the number of time frames of the behavior sequence; Step 12, the features of all nodes of M behavior subjects in N behavior sequences at all times are normalized, and the formula is: ； wherein, denotes a batch normalization operation; Step 13, for the behavior of two execution subjects in a behavior sequence, one frame is divided into two frames, and the divided result is integrated into a behavior sequence as a whole, and the formula is: 。 3. The human pose skeleton behavior recognition method based on adaptive graph convolution and self-attention according to claim 1, characterized in that, The step 2 comprises the following steps: Step 21, the adaptive graph convolution module reflects the relationship between different nodes in the human skeleton through three adjacency matrices Ak, Bk and Ck; where Ak represents the original normalized adjacency matrix; Bk represents adjacency matrix; Ck represents the data correlation graph; In order to determine whether there is a connection between two vertices and the strength of the connection, a normalized embedded Gaussian function is used to calculate the similarity of two vertices, and the formula is: ； in, Represents the total number of vertices; Represents vertices ; Represents vertices ; All indicate The embedding function for convolution operations; T represents the transpose operation, and e represents the natural constant. Represents vertices Features Transformed embedding vector; Represents vertices Features Transformed embedding vector; This indicates that for all vertices Summing the numerators; Step 22, the dot product is used to measure the similarity of two vertices in the embedding space.

4. The human pose skeleton behavior recognition method based on adaptive graph convolution and self-attention according to claim 3, characterized in that, The step 22 comprises the following steps: Step 221: Given the input feature map The size is By embedding functions and embedded functions Embedded into In the middle, the two embedded feature maps are rearranged and reshaped into Matrix and matrix; Step 222, to matrix and matrix multiplication, resulting in similarity matrix ; at the same time, normalizing the values of the similarity matrix between 0-1 to be used as soft edges for the two vertices: ； where and are parameters of the embedding functions and respectively, initialized to 0. Step 223, the obtained Ck, Bk and Ak are added, and the output of the adaptive convolution layer is obtained through the calculation of the weighting function, and the formula is: ； wherein denotes a weighting function.

5. The human pose skeleton behavior recognition method based on adaptive graph convolution and self-attention according to claim 1, characterized in that, The self-attention module generates query, key and value vectors to calculate the attention weight between global nodes, and finally aggregates global features.

6. The human pose skeleton behavior recognition method based on adaptive graph convolution and self-attention according to claim 1, characterized in that, The multi-scale time convolution module in the step 3 comprises four parallel branches; The four parallel branches comprise a first branch and a second branch; the first branch and the second branch are two time convolutions with different receptive fields, and the formula is: ； wherein, represents a branch output of the temporal convolution; represents a branch 1x1 convolution operation of represents a temporal convolution operation, represents the size of the branch convolution kernel; represents the dilation rate of the temporal convolution; represents the output of the spatial module, i.e. the output of the AAGC module; The four parallel branches further comprise a third branch; the third branch is a 1*1 convolution and a max pooling operation, and the formula is: ； wherein, represents the output of the max-pooling branch; represents a max-pooling operation; The four parallel branches further comprise a fourth branch; the fourth branch is a 1*1 convolution, and the formula is: ； wherein denotes the output of the 1x1 branch; The outputs of all branches are spliced in the channel dimension to obtain the output of the MS-TCN, and the formula is: ； wherein, denotes the output of the multi-scale temporal convolution module, denotes a channel concatenation operation.

7. The human pose skeleton behavior recognition method based on adaptive graph convolution and self-attention according to claim 1, characterized in that, The step 4 comprises the following steps: Step 41, for the input T-frame human skeleton sequence, respectively, into the Adaptive module and the Self-Attention module, extract the spatial features with local and global receptive fields combined with adaptive graph convolution and self-attention ; Step 42, results of spatial modeling an incoming temporal convolution module to model temporal dependencies between different frames; Step 43, after L layers of continuous spatial and temporal alternating modeling, the final output features will pass through a global average pooling layer and a fully connected layer in turn to obtain the behavior classification score.

Images