A human skeleton behavior recognition method based on self-attention graph convolution

Abstract

The application discloses a human skeleton behavior recognition method based on self-attention graph convolution, which comprises a spatial mixed module MGS combined with a graph convolution and self-attention, a spatial self-attention module TSSA specific to a time frame, and a multi-scale time convolution module MS-TCN. In the MGS, the local and global relationships between nodes are modeled by performing the graph convolution and self-attention operations in parallel, and the bidirectional interaction between the two branches is performed to realize the information complementation in the channel and spatial dimensions. The TSSA module uses the self-attention to learn the intraspatial relationship in a behavior sequence, and models the unique spatial features of a single-frame human skeleton. The MS-TCN module adopts a multi-branch design, and uses the time convolutions with different hole rates to expand the time receptive field. By combining the MGS, the TSSA and the MS-TCN, a MixGCN is further proposed, and remarkable performance is achieved on a challenging benchmark dataset NTU RGB+D60.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for human skeleton behavior recognition based on self-attention map convolution. Background Technology

[0002] Human behavior recognition is a widely studied computer vision classification task that plays a crucial role in human-computer interaction, video understanding, and virtual reality. This task involves data from various modalities, such as RGB, optical flow, and skeletons. Skeleton data, which represents the human body structure using 2D or 3D coordinates of multiple joints, is computationally efficient, robust to interference from complex backgrounds and varying lighting conditions, and can mitigate visual privacy violations at the data source level. Therefore, using human skeleton data for behavior recognition has attracted increasing attention. However, the lack of contextual information about image content in skeleton data also makes this task extremely challenging.

[0003] Early deep learning-based methods treated human joints as a set of independent features, organizing them into feature sequences or pseudo-images, and then feeding them into recurrent neural networks or convolutional neural networks to predict action labels. However, the human skeleton is an irregular graph, and these methods could not fully explore the intrinsic connections between human joints. In recent years, Spatial Temporal Graph Convolutional Networks (ST-GCN) pioneered the use of spatiotemporal graphs to model skeletal motion patterns, overcoming the shortcomings of traditional methods. In ST-GCN, each spatiotemporal graph convolutional module consists of a spatial graph convolution and a temporal convolution. The spatial graph convolution fuses features from different joints based on a manually defined sparse adjacency matrix. For temporal modeling, a one-dimensional temporal convolution is applied in parallel to each joint to model joint-specific motion patterns. Limited by the manually defined adjacency matrix, ST-GCN only considers other joints that have direct physical connections to the target joint, ignoring relationships between joints that are not naturally connected. For example, a "clapping" action is shown. Although there is no direct physical connection between the two hands, the relationship between the two hands should be highly related when clapping.

[0004] In recent years, based on the ST-GCN concept, related research has proposed learnable topologies to improve spatial modeling capabilities, effectively enhancing the performance of human skeleton-based behavior recognition models. However, most current methods only consider short-range node correlations, neglecting dependencies between distant nodes crucial for behavior recognition. For example, for actions like "waving" and "throwing," recognition can be achieved by focusing on local hand or arm movements; while for "walking," which involves the coordination of all joints in the body, global motion must be considered for accurate recognition. Therefore, models need to possess the ability to model both local and global joint dependencies.

[0005] At the same time, existing methods share a common problem: when spatially modeling skeleton sequences, they use the same topology for all time frames, ignoring the unique spatial structure of each time frame. Since the joints of the human skeleton in different time frames have unique spatial relationships that differ from those in other time frames, using a shared topology for all time frames is a suboptimal choice. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention proposes a human skeleton behavior recognition method based on self-attention map convolution, aiming to improve the accuracy of human skeleton behavior recognition while capturing the local and global relationships of the human skeleton and preserving the unique spatial features of a single frame.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for recognizing human skeleton behavior based on self-attention map convolution, the method is as follows:

[0009] A spatial hybrid module MGS that combines graph convolution and self-attention models the local and global relationships between nodes by performing graph convolution and self-attention operations in parallel, and performs bidirectional cross-branch interaction between two branches to achieve information complementarity in the channel and spatial dimensions.

[0010] A temporal frame-specific spatial self-attention module (TSSA) uses self-attention learning behavior sequences to learn intra-frame spatial relationships and model the unique spatial features of a single-frame human skeleton.

[0011] It also includes a multi-scale temporal convolution module, MS-TCN, which employs a multi-branch design to expand the temporal receptive field using temporal convolutions with different dilation rates.

[0012] Preferably, the identification method specifically includes the following steps:

[0013] Step 1: Process NTU RGB+D sequences containing K-type human skeletal behaviors. Process it;

[0014] Step 2: Construct a spatial relationship modeling module, which includes a spatial hybrid module MGS that combines graph convolution and self-attention and a time-frame-specific spatial self-attention module TSSA, to simultaneously model the local and global relationships of key points and the unique spatial features of a single-frame skeleton, respectively.

[0015] Step 3: Construct the multi-scale temporal convolution module MS-TCN

[0016] The MS-TCN consists of four parallel branches, including two temporal convolutions with different receptive fields, as expressed in equation (20):

[0017]

[0018] in Represents the output of branch i after time convolution, Conv i This represents a 1×1 convolution operation on branch i. K represents the temporal convolution operation. i ×1 represents the size of the convolution kernel in the i-th branch, d i Denotes the dilation rate of the convolution at time i, O spatial This indicates the output of the space module, including the outputs of the MGS module and the TSSA module;

[0019] The third branch consists of a 1×1 convolution and a max pooling operation, expressed as equation (21):

[0020] MP out =MaxPool[Conv(O spatial )] (twenty one)

[0021] MP out This represents the output of the max pooling branch, MaxPool represents the max pooling operation, and Conv represents the 1×1 convolution operation.

[0022] The fourth branch consists of only 1×1 convolutions, expressed as equation (22):

[0023] Out 1×1 =Conv(O spatial ) (twenty two)

[0024] Where Out 1×1 This represents the output of a 1×1 branch;

[0025] The output of the MS-TCN is obtained by concatenating the outputs of all branches along the channel dimension, as shown in equation (23):

[0026]

[0027] Among them O MS-TCN represents the output of the multi-scale temporal convolution module, and || represents the channel concatenation operation;

[0028] Step 4: Construct the MixGCN network

[0029] The MixGCN network mainly consists of a spatial hybridization module MGS, a spatial self-attention module TSSA, and a temporal convolution module MS-TCN.

[0030] For an input T-frame human skeleton sequence, it is fed into the Spatial Hybridization Module (MGS) and the Spatial Self-Attention Module (TSSA) respectively, to extract the spatial features with local and global receptive fields obtained by combining graph convolution and self-attention. MGS And the unique spatial features Z of each frame T ;

[0031] The output features of the spatial self-attention module TSSA are multiplied by a learnable coefficient β and then added to the output features of the spatial mixing module MGS, as expressed in equation (24):

[0032] O spatial =MGS(X)+β*TSSA(X) (24)

[0033] Where MGS(X) represents the skeleton behavior sequence features extracted by the spatial mixing module, and TSSA(X) represents the unique spatial features of the human skeleton in each single frame in the skeleton behavior sequence extracted by the spatial self-attention module.

[0034] Then output O of the space module spatial The input is a temporal convolution module used to model the temporal correlation between different frames, as expressed in equation (25):

[0035]

[0036] Among them O MS-TCN This represents the output of the multi-scale temporal convolution module;

[0037] After L layers of continuous alternating spatial and temporal modeling, the final output features will pass through a global average pooling layer and a fully connected layer in sequence to obtain the behavior classification score.

[0038] Preferably, the specific processing method in step 1 is as follows:

[0039] Step 1.1: Adjust the human skeleton behavior sequence for each category to 64 frames;

[0040] Step 1.2: Expand the corresponding features of different joints of M actors in N skeleton sequences at all times, as shown in Equation (1):

[0041]

[0042] Where X represents the input human skeleton behavior sequence, X.view(·) represents adjusting the shape of the input skeleton behavior sequence, N represents the number of human skeleton joints, V represents the number of joints, V=25 is used uniformly in this method, C is the input feature dimension, and T represents the number of time frames of the behavior sequence.

[0043] Step 1.3: Normalize the features of all key points of M actors in N action sequences at all times, as shown in Equation (2):

[0044]

[0045] Where BatchNorm(·) represents the batch normalization operation;

[0046] Step 1.4: For a behavior sequence with two executing entities, split one frame into two frames, and integrate the split results into a single behavior sequence, as shown in equation (3):

[0047]

[0048] Preferably, the specific processing method in step 2 is as follows:

[0049] Step 2.1: Construct a spatial hybrid module MGS that combines graph convolution and self-attention.

[0050] Step 2.2, Time-Frame-Specific Spatial Self-Attention Module (TSSA)

[0051] The Spatial Self-Attention (TSSA) module is used to model the unique spatial features of a single frame skeleton data. This is achieved first through two simple linear transformations. φ(·) transforms the input features into high-level features, as expressed in equations (16) and (17):

[0052]

[0053]

[0054] in And φ(·) is a 2D convolution with two kernels of size 1. Indicates the transformed features. The weight matrix is ​​represented by ; the model is initialized and trained and optimized using the backpropagation algorithm. Then, the correlation between nodes within a single time frame is calculated using the dot product method on the transformed features, as expressed in equation (18):

[0055] TA = Softmax(X) a X b '), (18)

[0056] in This represents the unique spatial relationship between joints within each frame of the input skeleton sequence T frames; Softmax represents the activation function used to calculate the attention score between nodes, and the ' symbol represents the transpose operation; then the input is multiplied by the obtained T relation matrices to model the spatial structure of the human skeleton in each frame, as shown in equation (19):

[0057]

[0058] Among them Λ T Represents aggregate functions, TA i X′ represents the intra-frame keypoint relationship in the i-th time frame. :,:,i This represents the transpose of the skeleton features at time i. The matrix inner product operation is represented by ||, and the concatenation operation is represented by ||. Within each channel, the spatial modeling results of each time frame are concatenated along the T-dimensional axis to obtain the final time frame-specific spatial modeling result.

[0059] Preferably, the specific processing method in step 2.1 is as follows:

[0060] Step 2.1.1: Construct the graph convolution module

[0061] The graph convolution module uses a sparse adjacency matrix A and a channel-specific topology CA to reflect the relationships between different joints in the human skeleton. A is manually defined based on the natural connectivity of the human skeleton and serves as a shared topology among all channels. CA is dynamically obtained based on the input skeleton features, as shown in Equation (4).

[0062] M(ψ(x i ),ω(x j ))=σ(Ρ(ψ(x i ))-Ρ(ω(x j (4)

[0063] Where x i and x jThese represent different skeleton key points. M(·) represents the correlation modeling function, ψ(·) and ω(·) represent two linear transformations used to reduce the dimensionality of the input features, σ(·) represents the activation function, and P(·) represents the average pooling and max pooling operations. Channel-specific correlations are obtained based on the correlation modeling function M(·). Where C' represents the number of output channels, the channel-specific correlation is combined with the shared topology to obtain the channel-angle topology, expressed as equation (5):

[0064] R = A + α * CA (5)

[0065] Where R represents the topology of the channel angle, α is a trainable scalar used to adjust the strength of the channel topology improvement, and the adjacency matrix A is added to each channel of α*CA. Then, the normalized skeleton features are transformed into a high-dimensional space through a linear transformation. Based on the obtained channel angle topology, the spatial features in each channel are extracted, as expressed in equations (6) and (7) respectively:

[0066] X'=ρ(X) (6)

[0067] Z G =Λ G (X',R)=[R1X′ :,1,: ||R2X′ :,2,: ||…||R C′ X′ :,C',: (7)

[0068] Where ρ(·) represents a linear transformation used to convert input features into high-level features, Λ G Represents aggregate functions, R i Describes the topology of a channel, X′ :,i,: This represents the feature of one channel of the input high-level features, and || represents the concatenation operation;

[0069] Step 2.1.2: Construct a self-attention module

[0070] The self-attention module is used to model the global dependencies between joints, which complements the local relationships modeled by graph convolution.

[0071] For input features that undergo two linear transformations in the spatial graph convolution module and Using a self-attention mechanism individually for each channel, the global relationships between joints within the channel are modeled, as expressed in Equation (8):

[0072]

[0073] in The softmax function represents the global relationships between keypoints within the i-th channel. represent the output features of the i-th channel after the input features have undergone two different linear transformations, and ' denotes matrix transpose;

[0074] By using the relation matrix SA i Multiplying the input features of the corresponding channel by the input features yields the global spatial feature modeling result of the channel angle, as shown in Equation (9):

[0075] Z S =Λ S (X',SA)=[(SA1)X' :,1,: ||(SA2)X′ :,2,: ||…||(SA C' )X′ :,C',: (9)

[0076] Z S Λ represents the output feature of the self-attention module. S SA represents the aggregation function, and SA represents the global interrelationships between all key points within all channels.

[0077] Step 2.1.3: Construct a feature fusion module

[0078] By designing a cross-branch bidirectional interaction between the graph convolution module and the self-attention module as described in steps 2.1.1 and 2.1.2, information complementarity in the channel dimension and spatial dimension is achieved.

[0079] The bidirectional interaction across branches includes, firstly, the channel interaction from the attention module to the graph convolution module, which includes average pooling operation, two consecutive 1×1 convolutions with normalization operations, between which the GELU activation function is used, and finally the channel attention result is generated by the Sigmoid function, as expressed in Equation (10):

[0080] C Attention =Sigmoid{BN2(Conv2[GELU[BN1(Conv1[avgpool(Z S (10)

[0081] Where C Attention This represents the complementary cues passed from the attention module to the graph convolution module; avgpool represents the average pooling operation; Conv1 and Conv2 represent 2D convolution operations; BN1 and BN2 represent normalization operations; and GELU represents the activation function.

[0082] The results of channel attention are applied across branches to the topology of channel angles to enhance the interrelationships between channels, as shown in Equation (11):

[0083] R Atten =R*C Attention (11)

[0084] Where R Atten The topology representing the channel angle after channel attention;

[0085] The spatial graph convolution module then uses the topological matrix that implements inter-channel interaction to spatially model the input features, as expressed in equation (12):

[0086]

[0087] Where Out G This represents the output of the graph convolutional module after receiving complementary cues from the self-attention module. This represents the topology of the channel angle of the i-th channel after channel attention;

[0088] Secondly, there is the spatial interaction between the graph convolution module and the self-attention module, which includes a pooling operation, a 1×1 convolution, and a sigmoid function, expressed as Equation (13):

[0089] S Atten =Sigmoid(Conv[avgpool(Out G (13)

[0090] Where S Atten The graph convolution module represents the spatial complementary cues passed to the self-attention module; avgpool represents the average pooling operation; and Conv represents the 2D convolution operation.

[0091] The modeling results obtained by applying this to the self-attention module are shown in Equation (14):

[0092] Out S =S Atten *Z S (14)

[0093] Where Out S This represents the output of the self-attention module after receiving complementary cues from the graph convolution module;

[0094] Finally, the modeling results of the two modules are added together to obtain the output of the spatial hybrid module, as shown in equation (15):

[0095] Out MGS =Out G +Out S (15)

[0096] Where Out MGSThis is the output of the spatial hybrid module. Existing technologies typically use only graph convolution or self-attention operations when extracting spatial features from skeleton data. This invention, for the first time, combines graph convolution and self-attention in the spatial modeling module. Based on CTR-GCN, this invention adds a self-attention operation, proposing a spatial hybrid module (MGS) that combines graph convolution and self-attention. On one hand, it combines graph convolution at the channel angle with self-attention to simultaneously model local and global relationships; on the other hand, it designs cross-branch interaction between the graph convolution and self-attention branches, achieving information complementarity between the two branches. Secondly, addressing the deficiency of general methods that use the same topological relationship to model skeleton sequences across all time frames, this invention designs a time-frame-specific spatial module (TSSA) based on self-attention, allowing the skeleton data of a single time frame to capture its own unique spatial features. Finally, this invention combines the proposed MGS, TSSA, and multi-scale temporal convolution module (MS-TCN) to design the MixGCN network.

[0097] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0098] 1) This invention uses a combination of graph convolution and self-attention to simultaneously model the local and global relationships between key points;

[0099] 2) This invention designs a cross-branch bidirectional interaction between graph convolution and self-attention parallel branches, achieving information complementarity in the channel and spatial dimensions respectively;

[0100] 3) This invention uses self-attention-independent modeling of time-frame-specific spatial relationships, which can capture the unique spatial features of a single frame of human skeleton. Attached Figure Description

[0101] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0102] Figure 1 This is a model structure diagram of the method proposed in this invention;

[0103] Figure 2 This invention visualizes several behaviors from the NTU RGB+D dataset.

[0104] Figure 3 This is a diagram of the spatial hybrid module structure combining graph convolution and self-attention proposed in this invention.

[0105] Figure 4This is a diagram of the cross-branch bidirectional interaction structure proposed in this invention;

[0106] Figure 5 This is a structural diagram of the time-frame-specific spatial self-attention module proposed in this invention;

[0107] Figure 6 This is the multi-scale temporal convolution module used in this invention. Detailed Implementation

[0108] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific embodiments.

[0109] This invention provides a method for human skeleton behavior recognition based on self-attention map convolution, such as... Figure 1-5 The procedure is as follows:

[0110] Step 1: Process NTU RGB+D sequences containing K (K=60) types of human skeletal behavior. Process it;

[0111] Step 1.1: Divide the human skeleton behavior sequence of each category into 64 segments, and randomly select one frame from each segment to form a new sequence of 64 frames;

[0112] Step 1.2: Expand the corresponding features of different joints of M actors in N skeleton sequences at all times, as shown in Equation (1):

[0113]

[0114] Where X represents the input human skeleton behavior sequence, X.view(·) represents adjusting the shape of the input skeleton behavior sequence, V represents the number of human skeleton joints, and V=25 is used uniformly in this method, C is the input feature dimension, and T represents the number of time frames of the behavior sequence.

[0115] Step 1.3: Normalize the features of all key points of M actors in N action sequences at all times, as shown in Equation (2):

[0116]

[0117] BatchNorm(·) represents the batch normalization operation.

[0118] Step 1.4: For a behavior sequence with two executing entities, split one frame into two frames, and integrate the split results into a single behavior sequence, as shown in equation (3):

[0119]

[0120] Step 2: Construct a spatial relationship modeling module, which includes a spatial hybrid module (MGS) combining graph convolution and self-attention and a time-frame-specific spatial self-attention module (TSSA), used to simultaneously model the local and global relationships of key points and the unique spatial features of a single-frame skeleton, respectively.

[0121] Step 2.1: Construct a spatial hybrid module (MGS) that combines graph convolution and self-attention;

[0122] Step 2.1.1: Construct the graph convolution module

[0123] The graph convolution module uses a sparse adjacency matrix A and a channel-specific topology CA to reflect the relationships between different joints in the human skeleton. A is manually defined based on the natural connectivity of the human skeleton and serves as a shared topology across all channels. CA is dynamically obtained based on the input skeleton features, as shown in equation (4):

[0124] M(ψ(x i ),ω(x j ))=σ(Ρ(ψ(x i ))-Ρ(ω(x j (4)

[0125] Where x i and x j These represent different skeleton key points. M(·) denotes the correlation modeling function, and ψ(·) and ω(·) represent two linear transformations used to reduce the dimensionality of the input features. σ(·) represents the activation function, and P(·) represents the average pooling and max pooling operations. Channel-specific correlations are obtained based on the correlation modeling function M(·). Where C' represents the number of output channels. Combining channel-specific correlation with the shared topology, the channel-angle topology is obtained, expressed as equation (5):

[0126] R = A + α * CA (5)

[0127] Where R represents the channel angle topology, α is a trainable scalar used to adjust the strength of the channel topology improvement, and the adjacency matrix A is added to each channel of α*CA. Then, the normalized skeleton features are transformed to a high-dimensional space through a linear transformation, and then, based on the obtained channel angle topology, the spatial features within each channel are extracted, expressed as equations (6) and (7) respectively:

[0128] X'=ρ(X) (6)

[0129] Z G =Λ G (X',R)=[R1X′:,1,: ||R2X′ :,2,: ||…||R C' X′ :,C',: (7)

[0130] Where ρ(·) represents a linear transformation used to convert input features into high-level features, Λ G Represents aggregate functions, R i Describes the topology of a channel, X′ :,i,: This represents the feature of one channel of the input high-level feature set, and || represents the concatenation operation. This method can effectively aggregate joint features from different channels.

[0131] Step 2.1.2: Construct a self-attention module

[0132] The self-attention module is used to model global dependencies between joints, complementing the local relationships modeled by graph convolution.

[0133] For input features that undergo two linear transformations in the spatial graph convolution module and Using a self-attention mechanism individually for each channel, the global relationships between joints within the channel are modeled, as expressed in Equation (8):

[0134]

[0135] in The softmax function represents the global relationships between keypoints within the i-th channel. ' represents the output feature of the i-th channel after the input feature has undergone two different linear transformations, and ' represents the matrix transpose.

[0136] By using the relation matrix SA i Multiplying the input features of the corresponding channel by the input features yields the global spatial feature modeling result of the channel angle, as shown in Equation (9):

[0137] Z S =Λ S =(X',SA)=[(SA1)X′ :,1,: ||(SA2)X′ :,2,: ||…||(SA C' )X′ :,C',: (9)

[0138] Z S Λ represents the output feature of the self-attention module. S SA represents the aggregation function, and SA represents the global interrelationship between all nodes within all channels.

[0139] Step 2.1.3: Construct a feature fusion module

[0140] By designing a cross-branch bidirectional interaction between the graph convolution module and the self-attention module as described in steps 2.1.1 and 2.1.2, information complementarity in the channel dimension and spatial dimension is achieved.

[0141] The bidirectional interaction across branches includes, firstly, the channel interaction from the attention module to the graph convolution module, which includes average pooling, two consecutive 1×1 convolutions with normalization, GELU activation function between them, and finally channel attention result generated by sigmoid. This is expressed as Equation (10):

[0142] C Attention =Sigmoid{BN2(Conv2[GELU[BN1(Conv1[avgpool(Z S )])]])} (10)

[0144] Where C Attention This represents the complementary cues passed from the self-attention module to the graph convolution module. avgpool represents the average pooling operation. Conv1 and Conv2 represent 2D convolution operations. BN1 and BN2 represent normalization operations. GELU represents the activation function.

[0145] The results of channel attention are applied across branches to the topology of channel angles to enhance the interrelationships between channels, as shown in Equation (11):

[0146] R Atten =R*C Attention (11)

[0147] Where R Atten This represents the topology of the channel angle after channel attention.

[0148] The spatial graph convolution module then uses a topological matrix that implements inter-channel interactions to spatially model the input features. This is expressed as equation (12):

[0149]

[0150] Where Out G This represents the output of the graph convolutional module after receiving complementary cues from the self-attention module. This represents the topology of the channel angle of the i-th channel after channel attention.

[0151] Secondly, the spatial interaction between the graph convolution module and the self-attention module consists of only one pooling operation, one 1×1 convolution, and one sigmoid function, expressed as equation (13):

[0152] SAtten =Sigmoid(Conv[avgpool(Out G (13)

[0153] Where S Atten The graph convolution module represents the spatial complementary cues passed to the self-attention module. Conv represents the average pooling operation and Conv represents the 2D convolution operation.

[0154] The modeling results obtained by applying this to the self-attention module are shown in Equation (14):

[0155] Out S =S Atten *Z S (14)

[0156] Where Out S This represents the output of the self-attention module after receiving complementary cues from the graph convolution module.

[0157] Finally, the modeling results of the two modules are added together to obtain the output of the spatial hybrid module, as shown in equation (15):

[0158] Out MGS =Out G +Out S (15)

[0159] Where Out MGS This is the output of the spatial mixing module.

[0160] Step 2.2, Time-Specific Spatial Self-Attention Module (TSSA)

[0161] The TSSA module is used to model the unique spatial features of a single frame skeleton data, first through two simple linear transformations. φ(·) transforms the input features into high-level features, as expressed in equations (16) and (17):

[0162]

[0163] X b =φ(X)=XW2 (17)

[0164] in And φ(·) is a 2D convolution with two kernels of size 1. Indicates the transformed features. The weight matrix is ​​initialized using the Kaiming method and continuously learned through backpropagation during network training. Then, the transformed features are calculated using a dot product to determine the correlation between nodes within a single frame, as shown in equation (18):

[0165] TA = Softmax(X) a X b '), (18)

[0166] in This represents the unique spatial relationship between joints in each of the T frames of the input skeleton sequence. Softmax represents the activation function used to calculate the attention score between nodes, and the ' symbol represents the transpose operation. Then, the input is multiplied by the obtained T relation matrices to model the spatial structure of the human skeleton in each frame, as shown in Equation (19):

[0167]

[0168] Among them Λ T Represents aggregate functions, TA i X represents the intra-frame keypoint relationship in the i-th time frame. : ' ,:,i This represents the transpose of the skeleton features at time i. The matrix inner product operation is represented by ||, and the concatenation operation is represented by ||. Within each channel, the spatial modeling results of each time frame are concatenated along the T-dimensional axis to obtain the final time frame-specific spatial modeling result. This module can capture unique spatial features within each time frame, overcoming the shortcomings of using a uniform topology across all time frames.

[0169] Step 3: Construct a multi-scale temporal convolutional module (MS-TCN)

[0170] The MS-TCN consists of four parallel branches, including two temporal convolutions with different receptive fields, as expressed in equation (20):

[0171]

[0172] in Represents the output of branch i after time convolution, Conv i This represents a 1×1 convolution operation on branch i. K represents the temporal convolution operation. i ×1 represents the size of the convolution kernel in the i-th branch, d i Denotes the dilation rate of the convolution at time i, O spatial This indicates the output of the space module, including the outputs of the MGS module and the TSSA module.

[0173] The third branch consists of a 1×1 convolution and a max pooling operation, expressed as equation (21):

[0174] MP out=MaxPool[Conv(O spatial )] (twenty one)

[0175] MP out The output represents the max pooling branch, MaxPool represents the max pooling operation, Conv represents the 1×1 convolution operation, and O represents the output of the max pooling branch. spatial This represents the output of the space module.

[0176] The fourth branch consists of only 1×1 convolutions, expressed as equation (22):

[0177] Out 1×1 =Conv(O spatial ) (twenty two)

[0178] Where Out 1×1 Represents the output of a 1×1 branch, Conv represents a 1×1 convolution operation, O spatial This indicates the output of the space module.

[0179] The output of the MS-TCN is obtained by concatenating the outputs of all branches along the channel dimension, as shown in equation (23):

[0180]

[0181] Among them O MS-TCN The output of the multi-scale temporal convolution module is represented by ||, and the channel concatenation operation is represented by ||.

[0182] Step 4: Construct the MixGCN network

[0183] The MixGCN network mainly consists of MGS, TSSA, and MS-TCN.

[0184] For the input T-frame human skeleton sequence, it is fed into the MGS module and the TSSA module respectively, and the spatial features with local and global receptive fields, which are obtained by combining graph convolution and self-attention, are extracted respectively. MGS And the unique spatial features Z of each frame T The output features of the TSSA are multiplied by a learnable coefficient β and then added to the output features of the MGS module, as shown in equation (24):

[0185] O spatial =MGS(X)+β*TSSA(X) (24)

[0186] Among them O spatialFor the overall output of the spatial module, MGS(X) represents the skeleton behavior sequence features extracted by the spatial fusion module, and TSSA(X) represents the unique spatial features of the human skeleton in each single frame within the skeleton behavior sequence extracted by the spatial self-attention module. β is a learnable coefficient used to adjust the importance of spatial relationships specific to a given time frame.

[0187] The result of the spatial modeling is then O spatial The input is a temporal convolution module used to model the temporal correlation between different frames, as expressed in equation (25).

[0188]

[0189] Among them O MS-TCN Represents spatial modeling results O spatial Input the output of the multi-scale temporal convolution module.

[0190] After L layers of continuous alternating spatial and temporal modeling, the final output features will pass through a global average pooling layer and a fully connected layer in sequence to obtain the behavior classification score.

[0191] Example:

[0192] The test results of the present invention are further described in conjunction with the following charts:

[0193] To verify the effectiveness of the method of this invention, experiments were conducted on the commonly used human skeleton behavior dataset NTU RGB+D 60. To ensure a fair comparison with other methods, the widely used Cross-Subject and Cross-View metrics were used as evaluation indicators, and the Top-1 accuracy was reported.

[0194] Table 1 shows the action recognition accuracy on the NTU RGB+D 60 dataset compared to existing methods.

[0195]

[0196]

[0197] As shown in Table 1, the proposed MixGCN achieves top-1 accuracy of 93.1% and 96.6% on the X-sub and X-view evaluation metrics, respectively. Compared to PSUMNet, which requires manually defining additional human skeleton data, this invention is entirely data-driven and still achieves 0.2% higher accuracy on the X-sub metric.

[0198] The experimental results in Table 1 show that the method of the present invention achieves the best results on the X-Sub standard compared with other methods, and achieves very competitive results with the latest methods on the X-View standard, proving the effectiveness and feasibility of the method proposed in this invention.

[0199] Meaning of proper nouns and English abbreviations

[0200]

[0201]

[0202] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for human skeleton behavior recognition based on self-attention map convolution, characterized in that, The identification method is as follows: A spatial hybrid module MGS that combines graph convolution and self-attention models the local and global relationships between nodes by performing graph convolution and self-attention operations in parallel, and performs bidirectional cross-branch interaction between two branches to achieve information complementarity in the channel and spatial dimensions. A temporal frame-specific spatial self-attention module (TSSA) uses self-attention learning behavior sequences to learn intra-frame spatial relationships and model the unique spatial features of a single-frame human skeleton. And a multi-scale temporal convolution module MS-TCN. The MS-TCN module adopts a multi-branch design and expands the temporal receptive field by using temporal convolution with different dilation rates. Specifically, the following steps are involved: Step 1: Process NTU RGB+D sequences containing K-type human skeletal behaviors. Process it; Step 2: Construct a spatial relationship modeling module, which includes a spatial hybrid module MGS that combines graph convolution and self-attention and a time-frame-specific spatial self-attention module TSSA, to simultaneously model the local and global relationships of key points and the unique spatial features of a single-frame skeleton, respectively. The Spatial Self-Attention (TSSA) module is used to model the unique spatial features of a single frame skeleton data. This is achieved first through two simple linear transformations. , The input features are transformed into high-level features, as expressed in equations (16) and (17): （16）； （17）； in as well as It is a 2D convolution with two kernels of size 1. , Indicates the transformed features. The weight matrix is ​​represented by ; the model is initialized and trained and optimized using the backpropagation algorithm. Then, the correlation between nodes within a single time frame is calculated using the dot product method on the transformed features, as expressed in equation (18): （18）； in This represents the unique spatial relationship between joints within each frame of the input skeleton sequence T frames; This represents the activation function used to calculate the attention score between nodes. The symbol represents the transpose operation; then the input is multiplied by the T relation matrices obtained to model the spatial structure of the human skeleton in each frame, as shown in equation (19): (19)； in Represents aggregate functions, This represents the intra-frame keypoint relationship in the i-th time frame. This represents the transpose of the skeleton features at time i. This represents the matrix inner product operation. This indicates a stitching operation, which stitches the spatial modeling results of each time frame in each channel along the T dimension to obtain the final time-frame-specific spatial modeling result. ; Step 3: Construct the multi-scale temporal convolutional module MS-TCN; Step 4: Construct the MixGCN network The MixGCN network mainly consists of a spatial hybridization module MGS, a spatial self-attention module TSSA, and a temporal convolution module MS-TCN. For the input T-frame human skeleton sequence, it is fed into the spatial blending module MGS and the spatial self-attention module TSSA, respectively, to extract spatial features with local and global receptive fields obtained by combining graph convolution and self-attention. and the unique spatial features of each frame ; Multiply the output features of the spatial self-attention module TSSA by a learnable coefficient. The output characteristics of the spatial mixing module MGS are then added together, as expressed in equation (24): （24）； in This represents the skeleton behavior sequence features extracted by the spatial mixing module. This represents the unique spatial feature of each single-frame human skeleton in the skeleton behavior sequence extracted by the spatial self-attention module; Then output the space module The input is a temporal convolution module used to model the temporal correlation between different frames, as expressed in equation (25): （25）； in This represents the output of the multi-scale temporal convolution module; After L layers of continuous alternating spatial and temporal modeling, the final output features will pass through a global average pooling layer and a fully connected layer in sequence to obtain the behavior classification score.

2. The identification method according to claim 1, characterized in that, The specific processing method in step 1 is as follows: Step 1.1: Adjust the human skeleton behavior sequence for each category to 64 frames; Step 1.2: Expand the corresponding features of the M action points of the N skeleton sequences at all times, as shown in Equation (1): （1）； in This represents the input sequence of human skeletal behaviors. This indicates adjusting the shape of the input skeleton behavior sequence, where N represents... The number of joints in the human skeleton is represented by the number of joints used in this method. , For the input feature dimension, Indicates the number of time frames in the action sequence; Step 1.3: Normalize the features of all key points of M actors in N action sequences at all times, as shown in Equation (2): （2）； in This indicates a batch normalization operation; Step 1.4: For a behavior sequence with two executing entities, split one frame into two frames, and integrate the split results into a single behavior sequence, as shown in equation (3): （3）。 3. The identification method according to claim 2, characterized in that, The specific processing method in step 2 is as follows: Step 2.1: Construct a spatial hybrid module (MGS) that combines graph convolution and self-attention; Step 2.2, Time-Frame-Specific Spatial Self-Attention Module (TSSA).

4. The identification method according to claim 3, characterized in that, The specific processing method in step 2.1 is as follows: Step 2.1.1: Construct the graph convolution module The graph convolution module uses a sparse adjacency matrix. and channel-specific topology To reflect the relationship between different joints in the human skeleton, among which Manually defined based on the natural connections of the human skeleton, serving as a shared topology among all channels. The skeleton features are dynamically obtained based on the input, as shown in equation (4): （4）； in and These represent different skeletal joints. This represents the correlation modeling function. and This represents two linear transformations used to reduce the dimensionality of the input features. This represents the activation function. Representing average pooling and max pooling operations, based on a correlation modeling function. Obtain channel-specific correlations ,in The number of output channels is represented by the channel-specific correlation, which is combined with the shared topology to obtain the channel-angle topology, expressed as Equation (5): （5）； in Topology representing channel angles, It is a trainable scalar used to adjust the strength of channel topology improvements, where the adjacency matrix... will be added to For each channel, the normalized skeleton features are then transformed into a high-dimensional space through a linear transformation. Based on the topology of the obtained channel angles, the spatial features within each channel are extracted, as expressed in equations (6) and (7), respectively: （6）； （7）； in This represents a linear transformation used to convert input features into higher-level features. Represents aggregate functions, This represents the topology of a channel. This represents the feature of one channel of the input high-level features. Indicates a splicing operation; Step 2.1.2: Construct a self-attention module The self-attention module is used to model the global dependencies between joints, which complements the local relationships modeled by graph convolution. For input features that undergo two linear transformations in the spatial graph convolution module and For each channel, a self-attention mechanism is used individually to model the global relationships between joints within the channel, as expressed in equation (8): （8）； in This represents the global relationships between the joints within the i-th channel. This represents the activation function. , These represent the output features of the i-th channel after the input features have undergone two different linear transformations. Indicates matrix transpose; By using the relation matrix Multiplying the input features of the corresponding channel by the input features yields the global spatial feature modeling result of the channel angle, as shown in Equation (9): （9）； in This represents the output features of the self-attention module. Represents aggregate functions, This represents the global relationships between all nodes within all channels; Step 2.1.3: Construct a feature fusion module By designing a cross-branch bidirectional interaction between the graph convolution module and the self-attention module as described in steps 2.1.1 and 2.1.2, information complementarity in the channel dimension and spatial dimension is achieved. The bidirectional interaction across branches includes, firstly, the channel interaction from the attention module to the graph convolution module, which includes average pooling operations and two consecutive operations with normalization. The convolutions are performed, with the GELU activation function applied between them, and finally the channel attention result is generated by the Sigmoid function, as expressed in equation (10): （10）； in This represents the complementary channel cues passed from the self-attention module to the graph convolution module. This indicates the average pooling operation. and This represents a 2D convolution operation. and Represents normalization operation, Indicates the activation function; The results of channel attention are applied across branches to the topology of channel angles to enhance the interrelationships between channels, as shown in Equation (11): （11）； in The topology representing the channel angle after channel attention; The spatial graph convolution module then uses the topological matrix that implements inter-channel interaction to spatially model the input features, as expressed in equation (12): （12）； in This represents the output of the graph convolutional module after receiving complementary cues from the self-attention module. This represents the topology of the channel angle of the i-th channel after channel attention; Secondly, there is the spatial interaction between the graph convolution module and the self-attention module, including a pooling operation and a... The convolution of and a sigmoid function are expressed as equation (13): （13）； in This represents the spatial complementary cues passed from the graph convolution module to the self-attention module. This indicates the average pooling operation. Indicates a 2D convolution operation; The modeling results obtained by applying this to the self-attention module are shown in Equation (14): （14）； in This represents the output of the self-attention module after receiving complementary cues from the graph convolution module; Finally, the modeling results of the two modules are added together to obtain the output of the spatial hybrid module, as shown in equation (15): （15）； in This is the output of the spatial mixing module.

5. The identification method according to claim 1, characterized in that, The specific processing method in step 3 is as follows: The MS-TCN consists of four parallel branches, including two temporal convolutions with different receptive fields, as expressed in equation (20): （20）； in This represents the output of branch i after time convolution. Represents branch i Convolution operation, This indicates a temporal convolution operation. This represents the size of the convolution kernel in the i-th branch. This represents the dilation rate of the convolution at time i. This indicates the output of the space module, including the outputs of the MGS module and the TSSA module; The third branch is The convolution and max pooling operations are combined, and are expressed as equation (21): （21）； in This represents the output of the max-pooling branch. Represents max pooling operation. represent Convolution operation; The fourth branch is only composed of The convolutional composition is expressed as equation (22): （22）； in express Branch output; The output of the MS-TCN is obtained by concatenating the outputs of all branches along the channel dimension, as shown in equation (23): （23）； in This represents the output of the multi-scale temporal convolution module. This indicates a channel splicing operation.

Images