Skeleton-based interaction recognition with fusion split graph convolutional networks

Abstract

The application relates to the technical field of interactive recognition, in particular to a fusion split graph convolutional network based on skeleton interactive recognition. The steps are as follows: a fusion split graph is constructed; an interactive attention coding module is combined with the fusion split graph to form a fusion split graph convolution; and a fusion split graph convolutional network is formed through a short-term dependence module, the fusion split graph convolution and a hierarchical guided attention module. The fusion split graph convolutional network based on skeleton interactive recognition provided by the application extracts relevant features between interactive behaviors through a short-term dependence module, a fusion split graph convolution and a hierarchical guided attention module; the fusion split graph convolution extracts features in the same semantic space by fusing node information of two persons into the same semantic space and splitting them into hierarchical sets, so that a fusion split graph is integrated; and the short-term dependence module extracts spatial and motion features of interactive behaviors.

Description

Technical Field

[0001] This invention relates to the field of interaction recognition technology, and in particular to a fusion split graph convolutional network for skeleton-based interaction recognition. Background Technology

[0002] As a popular research area, human-robot interaction recognition has broad application prospects in robotics, particularly in human-computer interaction. Robots can focus on human interactions, recognize human behavior, and also identify interactions between robots. In recent years, with the continuous development of deep learning, interaction recognition methods based on RGB and skeletal sequences have been proposed. However, RGB-based methods are affected by factors such as lighting and self-occlusion, making them ineffective for interaction recognition. Skeletal sequence-based interaction recognition methods, on the other hand, are unaffected by these factors and have achieved significant results.

[0003] Traditional skeleton-based interaction recognition methods treat human joints as independent features and construct feature sequences from them for action prediction using recurrent neural networks (RNNs) or convolutional neural networks (CNNs). However, these methods neglect the correlated features between intersecting joints. For example... Figure 1 As shown, the feature information between interactive behaviors is aggregated in the interaction part, making it difficult to extract their interaction information and distinguish the subtle differences between similar actions such as "shaking hands" and "high-fiving".

[0004] Meanwhile, Perez et al. proposed a two-stream interaction relationship network based on Long Short-Term Memory (LSTM) networks, called LSTM-I RN, to model the internal relationships between the same human joints and the joint relationships between different people.

[0005] However, LSTM-I RN does not consider the distance relationship between body parts and cannot learn the relevant features between two interacting parts.

[0006] Most existing single-person action recognition methods employ the concept of graph convolution. However, in action recognition, we often encounter scenarios involving both two-person interactions and single-person actions. Traditional graph convolution models typically treat two-person interactions as isolated entities, thus ignoring crucial interaction information between them;

[0007] To address this issue, we designed a skeleton-based interaction recognition fusion split graph convolutional network to provide an alternative technical solution. Summary of the Invention

[0008] Therefore, it is necessary to provide a skeleton-based interaction recognition fusion split graph convolutional network to address the aforementioned technical problems.

[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0010] A skeleton-based interaction recognition fusion split graph convolutional network, the steps of which are as follows:

[0011] Construct a fusion-split graph;

[0012] The interactive attention encoding module is combined with the fusion split graph to form a fusion split graph convolution;

[0013] A fused split graph convolutional network is formed by using a short-term dependency module, a fused split graph convolution, and a hierarchical guided attention module.

[0014] As a preferred embodiment of the skeleton-based interaction recognition fusion split graph convolutional network provided by the present invention, the steps for constructing the fusion split graph are as follows:

[0015] The joints of the two individuals are integrated into a unified whole, sharing the same feature space;

[0016] By selecting a starting node that exists in the same semantic space as its neighboring nodes, the entire joint is divided into different hierarchical sets;

[0017] Connect adjacent hierarchical node sets sequentially using edges.

[0018] As a preferred embodiment of the skeleton-based interaction recognition fusion split graph convolutional network provided by the present invention, the graph's hierarchical information expression is as follows:

[0019]

[0020] Among them, A i The adjacency matrix represents the adjacency of adjacent hierarchical sets. The adjacency matrix represents the entire graph structure; 2V represents the number of joint nodes for each person, C represents the channel dimension of the adjacency matrix, and L represents the number of layers.

[0021] The adjacency matrix is ​​constructed by sequentially connecting adjacent hierarchical node sets through different joint types to form an edge set, as shown in the following expression:

[0022] A i =θ i (H i →H i+1 ||H i+1 →H i ||H i +H i+1 )

[0023] Among them, H iH represents the set of the i-th hierarchical nodes. i →H i+1 Indicates from H i To H i+1 A set of edges.

[0024] As a preferred embodiment of the skeleton-based interaction recognition fusion split graph convolutional network provided by the present invention, the short-time dependency module is used to extract the spatial and motion features of the interaction behavior, and the steps are as follows:

[0025] For input Four copies were made, and multiple frames of time information T were subtracted from the beginning or end of the motion;

[0026] Dimension is represented as

[0027] Fill in zeros where dimensions do not match.

[0028] By concatenating the input features and short-term motion features, we obtain the spatial information and short-term motion information of the input features, and finally obtain the output Z as shown below:

[0029] Z = [(XX t-1 )||(XX t+1 ...||X].

[0030] As a preferred embodiment of the skeleton-based interaction recognition fusion split graph convolutional network provided by the present invention, the fusion split graph convolutional module is used to extract node features in each layer set and aggregate all layer outputs, and the steps are as follows:

[0031] Use linear transformation to transform input features Convert to high-level features;

[0032] Using correlation modeling functions Extract the spatial distribution features between two human joints under different movements, and use φ and ω to reduce the feature dimensionality;

[0033] Feature aggregation is performed on each channel, and the outputs of all hierarchical channels are parallelized to obtain H, as shown in the following expression:

[0034]

[0035]

[0036] H = [H1||H2||H3…||H L ]

[0037] Where σ(·) is the activation function; It is an operation of Einstein's summation convention;

[0038] Concatenate multiple output Hi values ​​along the channel dimension to obtain the final output H∈

[0039] As a preferred embodiment of the skeleton-based interaction recognition fusion split graph convolutional network provided by the present invention, the hierarchical guided attention module highlights the hierarchical set that is more relevant to the interaction behavior, and the steps are as follows:

[0040] Extract representative nodes from each level set;

[0041] After extracting the hierarchical information of representative nodes, each hierarchical feature does not share information;

[0042] The feature maps of all hierarchical information are multiplied by the convolution output feature map of the fused split map. The relevant hierarchical set with the highest weight of hierarchical information is selected by Euclidean distance, and then the output feature map Z is obtained by weighted summation by hierarchical dimension.

[0043] As a preferred embodiment of the skeleton-based interaction recognition fusion split graph convolutional network provided by the present invention, the extracted representative nodes are processed by a spatial pooling function. Enhanced, space pooling function The expression is as follows:

[0044]

[0045] Among them, L i v represents the number of key points in the i-th level set. j It is a representative node in the adjacent hierarchical set.

[0046] In a preferred embodiment of the skeleton-based interactive recognition fusion split-graph convolutional network provided by the present invention, the expression of the output feature map Z is:

[0047]

[0048] It is clear without a doubt that the technical solution described above in this application can solve the technical problem that this application aims to address.

[0049] Meanwhile, through the above technical solutions, the present invention has at least the following beneficial effects:

[0050] This invention provides a fusion split graph convolutional network for skeleton-based interaction recognition. By proposing a fusion split graph convolutional network, it can extract interaction information more effectively compared with other interaction recognition methods.

[0051] The Short-Term Dependency (STD) module, Fusion Split Graph Convolution (MSGC), and Hierarchical Guided Attention (HGA) module are used to extract relevant features between interaction behaviors. MSGC integrates a fusion split graph by fusing the node information of two people into the same semantic space and splitting them into hierarchical sets to extract features within the same semantic space. STD extracts the spatial and motion features of interaction behaviors, while HGA highlights the hierarchical features with high correlation between interaction behaviors.

[0052] MS-GCN has achieved state-of-the-art results on the NTU-RGB+D and NTU-RGB+D 120 interactive datasets, and also performs well in single-person action recognition based on skeleton data. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of existing technology;

[0055] Figure 2 This is the global split diagram of the present invention;

[0056] Figure 3 This is a diagram showing the overall architecture of the Fusion Split Graph Convolutional Network (MS-GCN) of the present invention;

[0057] Figure 4 This is a schematic diagram of the short-term dependency block of the present invention;

[0058] Figure 5 This is a visual illustration of the "handshake" and "high five" actions of the present invention;

[0059] Figure 6 This is a schematic diagram of the hierarchical attention scores of the "handshake" type HGA module of the present invention;

[0060] Figure 7 This is a schematic diagram comparing the accuracy of MS-GCN and CTR-GCN on the X-Set of the NTU-RGB+D 120 dataset of this invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0062] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0063] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0064] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0065] Figure 1 For the existing handshake and high-five interactions. In these two actions, "handshake" (a) and "high-five" (b), there is a semantic connection between the hands, and their subtle difference lies in the evolution of distance.

[0066] Reference Figures 2-7 A fusion split graph convolutional network for skeleton-based interaction recognition.

[0067] 1. Information recognition

[0068] 1.1 Behavior recognition based on graph convolutional networks

[0069] In skeleton-based action recognition, the human skeleton is constructed as a graph structure of joint nodes. To handle non-Euclidean data, graph convolution techniques have been proposed. Yan et al. proposed ST-GCN, which first uses a topological graph to extract and aggregate spatiotemporal information of the skeleton. However, ST-GCN uses a handcrafted skeleton graph and cannot extract relevant features between joints beyond natural connections.

[0070] Chen et al. further proposed CTR-GCN to simultaneously learn shared topology and channel correlations, learn unnatural connections between joint nodes, and enhance the feature extraction and aggregation capabilities of GCN without significantly increasing parameters. Liu et al. proposed Mask-GCN, which focuses on learning skeletal joints for specific actions. These joints primarily convey motion information, while masking skeletal joints that convey rare motion information and are more susceptible to novel motion patterns, in order to learn stable and information-rich motion features. In this work, the focus is on skeleton-based interaction recognition, and a modeling approach for human interaction is proposed, considering both semantic and motion information.

[0071] 1.2 Skeleton-based Two-person Interaction Recognition

[0072] With the development of deep learning, graph convolutional neural networks (GCNs) have been widely used in skeleton-based action recognition and interaction recognition. For example, Gao et al. proposed the Attention Interactive Graph Convolutional Network (AIGCN) to explore two-person interactions. They designed an interactive attention encoding GCN (IAE-GCN) module to extract the interactive spatial structure, while treating joint position encoding as semantic information to reflect the degree of their mutual influence.

[0073] Secondly, they used Interactive Attention Masking TCN (IAM-TCN) to extract temporal interaction features, representing temporal attentional excitation signals between different joints of different people. Li et al. proposed 2P-GCN, introducing a new unified two-person graph to represent the in-body and out-of-body interactions between joints, and designed several graph labeling strategies to supervise the model's learning to distinguish spatiotemporal interaction features. Some studies have also utilized pose estimation and keypoint detection techniques to extract features of human interaction behavior, in addition to using graph convolutional neural networks. For example, Chen et al. proposed a behavior recognition method based on pose estimation, first using a convolutional neural network to extract human poses, and then using LSTM to model the time series to obtain behavior representations. In addition, Liu et al. proposed an interaction behavior modeling method based on pose estimation and keypoint detection, employing a two-stage process including pose estimation and keypoint detection networks to extract features of interaction behavior. However, these methods neglect the correlation features between the interactions of two individuals. In contrast, fusing the interactions of two individuals into the overall interaction recognition, extracting their distance and semantic information at a global level, can capture their interaction features.

[0074] 1.3 Attention Mechanism for Action Recognition

[0075] Attention mechanisms were initially proposed in the field of Natural Language Processing (NLP) and are a fundamental element of deep neural networks. In recent years, attention networks such as SEnet proposed by Hu et al. have begun to introduce attention mechanisms into the field of computer vision. The SEnet network uses spatially oriented attention and a multilayer perceptron to extract channel information. Furthermore, Woo et al. proposed CBAM, which uses channel-oriented and spatially oriented attention. Given an intermediate feature map as input, the CBAM module infers an attention map along two independent dimensions (channel and space), and then multiplies the attention map with the input feature map for adaptive feature optimization. In the context of graph convolutional networks, attention mechanisms have been proposed to aggregate features from neighboring nodes. The weights of neighboring node features are determined solely by the node features and are independent of the underlying graph structure. In this study, we propose Hierarchical Guided Attention (HGA), which highlights hierarchical features that are more relevant in interactive behavior and learns subtle differences between interactions.

[0076] 2. Fusion Split-Graph Convolutional Networks

[0077] like Figure 3 As shown, the Fusion Split Graph Convolutional Network (MS-GCN) comprises a Short Temporal Dependency (STD) module, a Fusion Split Graph Convolutional (MSGC) module, and a Hierarchical Guided Attention (HGA) module. The STD module extracts spatial and motion features of the interaction behavior, the MSGC module extracts node features from each hierarchical set and aggregates all hierarchical outputs, and the HGA module highlights the hierarchical sets that are more relevant to the interaction behavior. A novel graph structure for interaction recognition, the Fusion Split Graph, is proposed. This proposed model is effective not only for interaction recognition but also for individual action recognition.

[0078] 2.1 Fusion and Splitting Diagram

[0079] Traditional methods for constructing human skeleton graphs use natural connections as edges, but lack information about the connections between joint nodes and boundary nodes. The fused split graph proposed in this paper is a state-of-the-art method for constructing human skeleton graphs. It merges the joint nodes of two individuals and divides them into multiple hierarchical sets in the semantic space. Figure 2 A framework for fusion split graphs is presented, which divides the two-person interaction graph into a single-person-single-two-person hierarchical graph and constructs three adjacency matrices.

[0080] The construction of the fusion split graph involves two key steps. First, the joints of the two individuals are merged into a unified whole, sharing the same feature space. Then, by selecting a starting node whose neighboring nodes exist in the same semantic space, the overall joint is segmented into different hierarchical sets. Choosing the abdominal node as the starting node, and expanding outwards sequentially, a hierarchical set is formed between the two individuals; 50 nodes will form 6 hierarchical sets. These starting nodes then expand outwards, forming different semantic spaces, making more efficient and accurate analysis of joint motion possible. The hierarchical node sets are formed by grouping nodes within the same semantic space, such as the hand and foot joints of the two individuals.

[0081] To construct the graph, adjacent hierarchical node sets are sequentially connected by edges, which represent the connection relationships between nodes in adjacent sets. The resulting fused split graph effectively extracts features that capture the evolution of joint interactions, distances, and joint motion characteristics. After sequential connections, the constructed graph structure contains the graph's hierarchical information, which is defined as... A i The adjacency matrix represents the adjacency of adjacent hierarchical sets. This represents the adjacency matrix of the entire graph structure. Note that 2V represents the number of joint nodes between the two individuals, C represents the channel dimension of the adjacency matrix, and L represents the number of layers. Mathematically, this process is expressed by the formula:

[0082] Ai =θ i (H i →H i+1 ||H i+1 →H i ||H i +H i+1 )

[0083] Among them, H i H represents the set of the i-th hierarchical nodes. i →H i+1 Indicates from H i To H i+1 A set of edges. Note that || represents a series of operations. For the adjacency matrix θ i Normalization is performed to prevent the inner product of the adjacency matrix and the feature matrix from altering the original feature distribution. The adjacency matrix of the graph formed by the i-th hierarchical set and the (i+1)-th hierarchical set is defined as A. i It does not contain connection information for other hierarchical sets.

[0084] To take into account the information of the entire global graph, the edge connections between each hierarchical set must be considered; therefore, the adjacency matrix A is formulated as follows:

[0085] Traditional methods for constructing human skeletal graphs use natural connections as edges and joints as nodes. However, when dealing with two-person interactions, simply constructing a single-person graph is insufficient to describe global information. To address this issue, this paper proposes a novel method, such as... Figure 1 As shown, the interaction between two people is transformed into three graphs: the first person's graph, the second person's graph, and the global graph.

[0086] A = [A1||A2||A3…||A L ]

[0087] Here, L represents the number of hierarchical sets. Through fusion and splitting, adjacency matrices with different semantic spatial information were constructed.

[0088] 2.2 Short-term dependency modules

[0089] Traditional interaction recognition based on graph convolutional networks typically focuses on long-term behavioral dependencies, neglecting crucial temporal and spatial information and lacking attention to short-term behavioral changes. However, short-term temporal and spatial changes are equally important for recognizing two-person behaviors, and extracting short-term changes in motion makes learning subtle differences between similar actions particularly crucial for interaction recognition. Therefore, as... Figure 4 As shown, a short-time dependency module is proposed to model short-term motion information and joint features in space and time.

[0090] To extract short-term motion information from spatiotemporal context, the input is first processed... Four copies were made, and one or two frames of temporal information T were subtracted from the beginning or end of the motion. Their dimensions can be represented as... Then, fill in the areas where the dimensions don't match with zeros. For example... Figure 4 As shown, they are subtracted in pairs and feature transformed to extract short-term spatiotemporal motion information. The input features and short-term motion features are concatenated to obtain the spatial and short-term motion information of the input features, resulting in the output Z shown below:

[0091] Z = [(XX t-1 )||(XX t+1 ...||X].

[0092] 2.3 Merging Split Graph Convolution

[0093] Fusion Split-Graph Convolutional Network (MSGC) Figure 3 As shown. First, a linear transformation is used to transform the input features. This is transformed into advanced features. To prevent data redundancy, temporal features are averaged, with a focus on spatial features, and then a correlation modeling function is used. Spatial distribution features between two human joints under different movements are extracted, and φ and ω are used to reduce the feature dimensionality. The hierarchical adjacency matrix is ​​then used. Add to the joint feature to obtain the joint (x) i ,x j The pairwise correlations between them. It is worth noting that the adjacency matrix... is a learnable parameter, and unlike other graph convolutional networks, it places edge connections in one dimension. Finally, feature aggregation is performed on each channel, and the outputs of all layers and channels are parallelized to obtain H.

[0094]

[0095]

[0096] H = [H1||H2||H3…||H L ]

[0097] Here, α is a trainable scalar used to adjust the refinement strength, preventing the edge connection strength of the adjacency matrix A from being too high, which would prevent the learning of subtle differences between interactions. Input features These are high-level features that have undergone transformation. σ(·) is the activation function. Subtraction essentially calculates φ(x). i ) and ω(x jThe distance along the channel dimension was calculated, and spatial information between nodes was extracted. A correlation modeling function was used. This is used to capture the spatial distribution information of connection points between channels. Then, the spatial distribution features between joints are added to the hierarchical adjacency matrix Ai, and the edge connection information between hierarchical node sets is added to the features. This is an operation based on the Einstein summation convention. It aggregates the input features and each channel graph to obtain the output features of a single-level set. Concatenate these output Hi values ​​along the channel dimension to obtain the final output.

[0098] 2.4 Hierarchical guidance of attention

[0099] MSGC uses an aggregation strategy to sum all hierarchical outputs. However, since interactive behaviors can create correlations between keypoints and edge keypoints, a Hierarchical Guided Attention (HGA) module is proposed, which applies a weighted summation strategy along the hierarchical dimension to give appropriate attention to the hierarchical outputs.

[0100] The HGA module is used in feature maps after the MSGC layer. Above. To ensure an appropriate attention score, representative nodes are first extracted from each level set, as the number of edges connected to each node can vary significantly. This representative node extraction step is crucial for obtaining an accurate attention score. Once extracted, these representative nodes are spatially pooled to further enhance the effectiveness of the method. Spatial pooling function. The expression is:

[0101]

[0102] Among them, L i v represents the number of key points in the i-th level set. j It is a representative node in the adjacent hierarchical set. A pooling operation is performed on it along the time dimension T.

[0103] After extracting the hierarchical information of representative nodes, each hierarchical feature does not share information. Similarity in the hierarchical feature space is learned through Euclidean distance, and the feature maps of all hierarchical information are multiplied with the MSGC output feature map to obtain the output feature map obtained by weighted summation across the hierarchical dimensions. The expression for the output feature map Z is:

[0104]

[0105] After concatenating the outputs H(i) of each layer set in the MSGC layer along one dimension, the feature information of each layer set is contained. The feature similarity between each layer set is learned through Euclidean distance B, and after extraction, the feature similarity of each layer set is concatenated along the hierarchical dimension (i∈L). Then, the hierarchical information is passed through the Sigmoid activation function, which assigns adaptive weights to the features of each layer set and multiplies them with the input feature map to obtain the final output features.

[0106] 3. Experiment

[0107] 3.1 Experimental Dataset

[0108] The NTU-RGB+D (NTU-60) dataset comprises 40,320 motion segments across 60 motion categories performed by 25 actors from three different perspectives. It is currently one of the largest motion recognition datasets based on a single depth sensor. Each sample in this dataset includes a sequence of 3D human joint positions recorded by a depth sensor. The motions in this dataset include common everyday and athletic movements such as running, walking, and bending. The NTU-60 dataset contains 112 categories of two-person interactive motions; this portion of the dataset is referred to as NTU-60*.

[0109] The NTU-RGB+D120 (NTU-120) dataset is an expanded version of the NTU-RGB+D dataset, containing 118,081 motion segments from 120 motion categories. Similar to the NTU-RGB+D dataset, it is based on a single depth sensor and includes 3D human joint position sequences. However, the NTU-RGB+D120 dataset is more diverse and complex, featuring a wider range of human poses, movements, and environmental conditions. The NTU-120 dataset includes 26 two-person interaction motion categories; this portion of the dataset is referred to as NTU-120*.

[0110] The baseline model is CTR-GCN, which proposes a novel channel topology-optimized graph convolution that dynamically learns different topologies and efficiently aggregates joint features from different channels in skeleton-based action recognition.

[0111] In the experiments, the batch size was set to 16, and the CTR-GCN data preprocessing method was used. The model was trained using SGD with 0.9 momentum and 0.0004 weight decay. The training epochs were set to 65, with a warm start strategy used for the first five epochs to stabilize the training process. The learning rate adopted a dynamic decay strategy with a maximum value of 0.1 and a minimum value of 0.0001. All experiments were conducted on an RTX5000.

[0112] 3.2 Ablation Experiment

[0113] Table 1: Ablation Experiment Results of Fusion Split Graph Convolution

[0114]

[0115] Extensive ablation studies were conducted on the NTU-120* dataset to validate the effectiveness of the proposed Fusion Split Graph Convolution (MSGC), Short-Term Dependency Module (STD), and Hierarchical Guided Attention Module (HGA). In the ablation studies, only joint modality data was used, and a cross-subject (X-Sub) approach was employed. The results of the ablation studies are shown in Table 1.

[0116] First, the effectiveness of the fused split graph convolution was verified. The two-person interaction was treated as two single-person interactions, and the splitting idea was only used during graph construction. It was found that the splitting idea is equally effective in handling two-person interactions, improving accuracy by 0.65%. Compared to a single split graph, the proposed fused split graph improves accuracy by 1.01%, and compared to using a single-person split graph, it improves accuracy by 0.36%. It was observed that the fused splitting idea is more effective at extracting interaction information than the single splitting idea.

[0117] The visualization results of the fusion split diagram are as follows Figure 5 The diagram shows visualizations of the "handshake" and "high-five" actions. Hierarchical adjacency matrices for the second, fourth, and sixth hierarchical sets were selected. (a), (b), and (c) show the corresponding connections between the hand joints and all other bijoints under the hierarchical adjacency matrix for the "handshake" action, while (d), (e), and (f) correspond to the "high-five" action.) The adjacency matrices were set as learnable parameters, and adjacency matrices for the second, fourth, and sixth hierarchical sets were selected. For both the "handshake" and "high-five" actions, features are primarily clustered around the hand, reflecting the evolution of distance and joint features during the movement.

[0118] The connections between hand nodes and all other joints are selected, with the color intensity indicating the strength of the connection. A color value closer to zero indicates a weaker connection. In a handshake, there are clear interactive features between the hands, and the relevant connections between hand joint nodes are included in layers 4(b)(e) and 6(c)(f). Compared to (a) and (d), the connection strength between the hand and the interactive parts is higher, indicating that the fused split-graph convolutional network can learn the relevant features between interactive joints.

[0119] Next, the effectiveness of the Short-Term Dependency (STD) module and the Hierarchical Guided Attention (HGA) module in fusing the split graph was verified. As shown in Table 1, the STD module improved accuracy by 0.48% and the overall network model accuracy by 1.49%. This fully demonstrates the importance of extracting short-term spatiotemporal information for better learning subtle differences between similar actions. The HGA module improved the accuracy of the fused split graph by 0.51% and the overall model accuracy by 1.52%. For different actions, the required level of information to focus on is different. The HGA module assigns adaptive weights to each set of layers, learning the dependencies between each set of layers. Figure 6 As shown, for the "handshake" action, feature information clusters around the hand, and the hierarchical set containing nodes related to the hand scores significantly higher than the hierarchical set not containing such nodes. The HGA module can highlight the hierarchical set that is more relevant to the interaction behavior. Both the STD and HGA modules are found to be very effective.

[0120] Finally, the fusion split graph, short-term dependency (STD) module, and hierarchical guided attention (HGA) module were integrated into the network, resulting in a significant 1.92% improvement in accuracy compared to the baseline model. This result confirms the effectiveness of the MS-GCN model in capturing spatiotemporal dependencies and hierarchical relationships between learned actions.

[0121] Extensive ablation studies were conducted on the NTU-120* dataset to validate the effectiveness of the proposed global segmentation graph convolution and hierarchical aggregation attention module (HAA). In the ablation studies, only joint modality data were used, and the experimental results are shown in Table 1.

[0122] First, the effectiveness of the Fusion-Split-Graph Convolution (GSGC) was verified. The interaction between two people is treated as the interaction between two separate people, and only the segmentation idea is used when constructing the graph. It can be found that the segmentation idea is also effective in handling two-person interactions, improving accuracy by 0.35%. Compared with single-person segmentation graph convolution, the proposed global segmentation graph convolution improves accuracy by 0.85%, and compared with using single-person segmentation graph convolution by 0.5%. It can be seen that the "global segmentation" idea is more effective at extracting interaction information than the "single-person segmentation" idea.

[0123] Table 2: Experimental comparison between fused split-graph convolutional networks and state-of-the-art methods

[0124]

[0125] 3.4 Comparison with skeleton-based action recognition

[0126] Although the primary task of MS-GCN is two-person interaction recognition, the model was briefly validated on the X-Sub subset of the NTU-120 dataset, which includes not only two-person interactions but also single-person actions. Figure 7 As shown in the comparison (accuracy comparison of MS-GCN and CTR-GCN on the X-Set of the NTU-RGB+D120 dataset, involving 120 categories. Pink text indicates that CTR-GCN has higher accuracy in the action recognition category than our model, and green text indicates that our MS-GCN has higher accuracy in the action recognition category than CTR-GCN.), the comparison results show that our method has higher accuracy than CTR-GCN in 71 of the 120 action categories, especially in the recognition of interactive behaviors. For example, the accuracy of "taking a photo" and "following" is significantly higher than that of CTR-GCN. In addition, our method also shows significant advantages in single-person action recognition, such as "typing", "playing with a mobile phone", and "making a victory sign", with higher accuracy than CTR-GCN. MS-GCN is able to effectively learn the relevant features of interactive behaviors. Overall, compared with CTR-GCN, our method performs better on the entire NTU-120 dataset, which confirms the effectiveness of the proposed MS-GCN.

[0127] 3.5 Comparison with advanced technologies and methods

[0128] Many state-of-the-art methods employ a multi-stream fusion framework. For a fair comparison, the same framework was used. Specifically, results from two modalities were fused: joints and skeletons. Experimental results for the interaction categories on the NTU-RGB+D and NTU-RGB+D120 datasets are shown in Table 2. The proposed MS-GCN performs exceptionally well in human skeleton-based interaction recognition. Thanks to the proposed MSGC, STD, and HGA modules, MS-GCN outperforms CNN- and RNN-based methods. In the joint modality, MS-GCN achieves the highest interaction recognition accuracy on both the X-sub and X-set of NTU-RGB+D, improving upon CR-GCN by 1.38% and 1.56%, respectively. On the X-sub and X-set of NTU-RGB+D120, MS-GCN improves performance by 1.89% and 1.67%, respectively. In the case of two-modal fusion, namely the fusion of joint and skeleton modalities, MS-GCN improves performance by 1.84% and 0.86% on X-sub and X-set of NTU-RGB+D, and by 1.12% and 1.87% on X-sub and X-set of NTU-RGB+D120, respectively, compared with 2s-AIGCN. These results indicate that the proposed method can better represent the interaction characteristics of the two skeletons in space and time.

[0129] 4. Summary

[0130] A Fusion Split Convolutional Network (MS-GCN) is proposed, consisting of three modules: Short Temporal Dependencies (STD), Fusion Split Convolutions (MSGC), and Hierarchical Guided Attention (HGA). Furthermore, for interaction recognition, a fusion split graph is proposed to capture and learn interaction information. Extensive experimental results demonstrate that the proposed model achieves superior performance on the NTU-60 and NTU-120 interaction datasets. This model effectively captures spatiotemporal dependencies and hierarchical relationships between actions, improving the accuracy of interaction recognition. Future work will extend our model to cover multi-person motion, expanding its scope beyond interaction recognition.

[0131] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A fusion-split graph convolutional network for skeleton-based interaction recognition, characterized in that, The steps are as follows: Construct a fusion-split graph; The interactive attention encoding module is combined with the fusion split graph to form a fusion split graph convolution; A fused split graph convolutional network is formed by using a short-term dependency module, a fused split graph convolutional network, and a hierarchical guided attention module; The steps for constructing the fusion split graph are as follows: The joints of the two individuals are integrated into a unified whole, sharing the same feature space; By selecting a starting node that exists in the same semantic space as its neighboring nodes, the entire joint is divided into different hierarchical sets; Connect adjacent hierarchical node sets sequentially using edges; The hierarchical information expression of the graph is as follows: ； in, The adjacency matrix represents the adjacency of adjacent hierarchical sets. Represents the adjacency matrix of the entire graph structure; This represents the number of joint nodes for the two individuals. The channel dimension represents the adjacency matrix. Represents the number of layers; The adjacency matrix is ​​constructed by sequentially connecting adjacent hierarchical node sets through different joint types to form an edge set, as shown in the following expression: in, Indicates the first A hierarchical set of nodes, Indicates from arrive A set of edges; The short-term dependency module is used to extract the spatial and motion features of the interaction behavior, and the steps are as follows: For input Four copies were made, and multiple frames of time information T were subtracted from the beginning or end of the motion; Dimension is represented as , , , ; Fill in zeros where dimensions do not match. By concatenating the input features and short-term motion features, we obtain the spatial information and short-term motion information of the input features, and finally obtain the output Z as shown below: ； The fusion split graph convolution is used to extract node features from each hierarchical set and aggregate all hierarchical outputs. The steps are as follows: Input features are transformed by linear transformation Convert to high-level features; Through correlation modeling function Extract the spatial distribution features between two human joints under different movements, and use... and To reduce feature dimensionality; Feature aggregation is performed on each channel, and the outputs of all hierarchical channels are parallelized to obtain H, as shown in the following expression: in, It is an activation function; This is an operation based on Einstein's summation convention; The final output is obtained by concatenating multiple output Hi values ​​along the channel dimension. ; The hierarchical guidance attention module highlights the hierarchical set that is more relevant to the interaction behavior, as follows: Extract representative nodes from each level set; After extracting the hierarchical information of representative nodes, each hierarchical feature does not share information; The feature maps of all hierarchical information are multiplied by the convolutional output feature map of the fused split map. The set of relevant hierarchical information with the highest weights is selected by Euclidean distance, and then the output feature map is obtained by weighted summation by hierarchical dimension. ; The extracted representative nodes are then processed using a spatial pooling function. Enhanced, space pooling function The expression is as follows: in, Indicates the first The number of key points in each hierarchical set It is a representative node in the adjacent hierarchical set; The output feature map The expression is: 。

Images