Two-person interaction behavior recognition method based on time-decoupled hierarchical graph convolution

Abstract

The application provides a two-person interaction behavior recognition method based on time-decoupled hierarchical graph convolution, comprising the following steps: S1: taking optimized joint node data as the input of a network, constructing a hierarchical graph topology to obtain a hierarchical relationship matrix; S2: constructing a time-decoupled hierarchical graph convolution based on the topology optimization of a CTR-GC channel graph convolution, adopting the hierarchical relationship matrix obtained in S1, and constructing a THGC graph convolution block capable of simultaneously capturing time channel information and space channel information; S3: constructing a BLOCKGC based on the graph convolution in the THGC, and then constructing a TBGC graph convolution block; S4: stacking the TBGC graph convolution block and a multi-scale time convolution block to obtain a TB-GCN graph convolution network; and S5: performing deep feature extraction and classification on the input human skeleton data by using the TB-GCN graph convolution network to obtain a two-person interaction behavior recognition result. The method can effectively improve the accuracy of two-person interaction behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a two-person interaction behavior recognition method based on time-decoupled hierarchical graph convolution. BACKGROUND

[0002] With the rapid development of intelligent devices and sensing technology, two-person interaction behavior recognition based on skeleton data is increasingly important in many application scenarios, especially in the fields of monitoring, human-computer interaction, virtual reality and sports analysis. The skeleton data of the human body is captured by a depth camera or an inertial measurement unit, which can effectively reflect the key points and motion trajectories of the human body, providing rich information for understanding and analyzing human interaction behavior.

[0003] Skeleton data can effectively represent the relative position and angle relationship between each joint, which is crucial for understanding individual posture and human-to-human interaction. Lee et al. proposed a hierarchical decomposition graph convolutional network (HD-GCN) architecture, which introduces a hierarchical decomposition graph (HD-Graph) concept. HD-GCN effectively divides each joint node into multiple sets to extract the main structural adjacency edges and remote edges, and uses these edges to build HD-Graphs containing them, which are located in the same semantic space of the human skeleton. A hierarchical aggregation attention module is introduced to highlight the integration of key hierarchical edges in the HD-Graph. Although such HD-GCN effectively extracts local features of points, HD-GCN needs to handle complex graph structures, especially in the hierarchical decomposition of human skeleton models, which significantly increases the computational load, resulting in high time cost for training and inference. Liu et al. proposed a time-decoupled graph convolutional network (TD-GCN), which applies different adjacency matrices to skeleton data from different frames. In each convolutional layer of TD-GCN, high-level spatio-temporal features are extracted from skeleton data to obtain deep spatio-temporal information. Channel-dependent and time-dependent adjacency matrices corresponding to different channels and frames are calculated to capture the spatio-temporal dependence between skeleton joints. The spatio-temporal features of skeleton nodes and the topological information of adjacent skeleton nodes are fused using channel-dependent and time-dependent adjacency matrices. TD-GCN captures spatio-temporal features through channel and time dependence, but the omission of key information in the design and selection of adjacency matrices affects the recognition effect. Zhou et al. proposed to use the advantages of graph distance to describe the physical topology, thereby encoding the bone connection. By combining the topological representation of specific actions with continuous homology analysis to describe the system dynamics, important topological nuances that are often lost in traditional graph convolutional networks are effectively preserved. The redundancy problem of existing traditional graph convolutional networks in multi-relation modeling is revealed, and an effective improved graph convolution method (BlockGC) is proposed to solve the redundancy problem. Although this BlockGC reduces the number of parameters, BlockGCN may overfit under the condition of tight training data. Li et al. input the two-person joint data as a whole into the network, construct a two-person graph that can preserve the basic interaction relationship, and propose four hand-labeled strategies to further enhance the interaction relationship between two people. Due to the significant differences in connection relationships between different actions, the model is difficult to adapt to complex interaction scenarios.

[0004] In summary, existing two-person interaction behavior recognition algorithms based on joint data have poor adaptability, insufficient modeling of temporal continuity of behavior, and cannot effectively capture subtle changes over time. Hand-designed features often cannot fully capture interaction relationships and miss key dynamic information. Existing methods are slow, making two-person interaction behavior recognition based on joint data still a challenging research topic.

[0005] In summary, it is very meaningful to provide a two-person interaction behavior recognition method suitable for identifying complex interaction behaviors. SUMMARY

[0006] Therefore, the two-person interaction behavior recognition method based on time-decoupled hierarchical graph convolution is provided to solve the problems of the existing recognition methods, such as insufficient modeling of the time continuity of behaviors, inability to effectively capture subtle motion changes occurring in time, poor real-time performance, and low accuracy in interaction behavior recognition.

[0007] The technical scheme of the present application is as follows: the two-person interaction behavior recognition method based on time-decoupled hierarchical graph convolution comprises:

[0008] S1: The optimized joint node data is used as the input of the network, a hierarchical graph topology is constructed to obtain a hierarchical relationship matrix;

[0009] S2: The CTR-GC channel topology optimized graph convolution is constructed as a time-decoupled hierarchical graph convolution, the hierarchical relationship matrix obtained in S1 is used to construct a THGC graph convolution block capable of simultaneously capturing time channel information and spatial channel information;

[0010] S3: The graph convolution in the THGC is constructed as a BLOCKGC, and then as a TBGC graph convolution block;

[0011] S4: The TBGC graph convolution block and a multi-scale time convolution block are stacked to obtain a TB-GCN graph convolution network;

[0012] S5: The TB-GCN graph convolution network is used to perform deep feature extraction and classification on the input human skeleton data to obtain a two-person interaction behavior recognition result.

[0013] Preferably, the optimized joint node data in S1 is joint node data after noise removal and frame loss, and the optimization processing method is as follows: the joint node data of two persons is obtained through a depth camera Kinect v2, the obtained joint node data is subjected to noise removal and frame loss filtering, three strategies of calculating the frame length, dispersion degree, and motion value of the node data are used to remove noise and filter frame loss of the joint node data of two persons, a rooted tree is constructed with the abdominal node as the center, the human body joint nodes are divided into five layers, and the graph is divided so that the nodes in the same hierarchical structure edge set are stored in the same semantic space.

[0014] Preferably, the THGC graph convolution block constructed in S2 is composed of a residual structure of two branches and a 1*1 convolution, wherein one branch further extracts more detailed spatial feature information through a 1*1 convolution, a time pooling module, and processing of the obtained spatial features using self-contrast subtraction and a tanh activation function, and adjusting the channel number through a 1*1 convolution to obtain spatial features;

[0015] The other branch is composed of a bottleneck block composed of two flow branches, one of which extracts more detailed temporal feature information through a spatial pooling module, processes the obtained temporal features using self-contrast subtraction and a tanh activation function, and the other adjusts the channel number through a 1*1 convolution to retain its original feature information, the two flow features are fused to obtain temporal features, and the obtained temporal features are multiplied and added with the spatial features obtained in the previous branch and the weighted hierarchical relationship matrix to obtain spatio-temporal features.

[0016] Preferably, in S3, the graph convolution in THGC is constructed as BLOCKGC, and then as TBGC graph convolution block, including: processing the input data by standardizing the weight, introducing the relative position coding into the graph convolution, generating the k-hop graph distance coding and calculating the weight, performing batch normalization and using a 1*1 convolution as a residual connection of the down-sampling layer, and processing the result output through a ReLU activation function.

[0017] The application provides a two-person interaction behavior recognition method based on time-decoupled hierarchical graph convolution, which can effectively solve the problem of insufficient modeling of time continuity in interaction behavior recognition, solve the problem of poor real-time performance of interaction behavior recognition of large data, and effectively improve the accuracy of complex interaction scenarios, greatly improving the efficiency of two-person interaction behavior recognition. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows, and obviously, other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A two-person interaction behavior recognition method based on time-decoupled hierarchical graph convolution is provided.

[0021] Figure 2 A multi-scale time convolution block structure schematic diagram provided for the disclosed embodiment of the present application;

[0022] Figure 3 A TBGC graph convolution block structure schematic diagram provided for the disclosed embodiment of the present application;

[0023] Figure 4 A hierarchical topology structure schematic diagram provided for the disclosed embodiment of the present application;

[0024] Figure 5 A TB-GCN graph convolution network overall structure diagram provided for the disclosed embodiment of the present application;

[0025] Figure 6 A result confusion matrix of NTU RGB+D interaction data provided for the disclosed embodiment of the present application. DETAILED DESCRIPTION

[0026] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, the same numbers refer to the same or similar elements throughout the drawings. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they only represent examples of systems consistent with some aspects of the present application, as detailed in the appended claims.

[0027] In view of the problems in the prior art two-person interaction behavior recognition method based on joint node data, the embodiment provides a two-person interaction behavior recognition method based on time decoupling hierarchical graph convolution, so as to effectively improve the accuracy of two-person interaction behavior recognition. The overall framework flowchart is as shown in Figure 1 Specifically, the following steps are included:

[0028] S1: Taking the joint node data as the network input, processing the noise and lost frames in the input joint node data to obtain the optimized joint node data as the network input, and constructing the hierarchical graph topology through the optimized joint data to obtain a hierarchical relationship matrix capable of simultaneously capturing the natural connection and non-natural connection of the human body;

[0029] The processing of the noise and lost frames in the input joint node data to obtain the optimized joint node data includes: obtaining the two-person joint node data by using a depth camera Kinect v2, calculating the frame length of the joint node data and filtering out the shorter skeleton data, identifying the dispersion degree of the joint nodes on the plane and removing the frames containing too much noise, and calculating the motion value to screen out the skeleton data whose motion amplitude is not within the predetermined range, so as to obtain the optimized joint node data.

[0030] The optimized joint data is taken as an input of a time decoupling hierarchical graph convolutional network model, and the optimized joint data is used to construct a hierarchical graph topology. Specific steps of constructing the hierarchical graph topology are as follows:

[0031] 1) The graph is first decomposed through the physical connection edges of the joints, a rooted tree is constructed, and the given joint is decomposed into the rooted tree;

[0032] 2) The abdominal node is taken as a centroid, the nodes in the same hierarchical edge set are distributed in the same semantic space, and a directed adjacency matrix of H hierarchical edge sets with L levels is defined as As shown in formula (1), wherein H k represents the kth hierarchical node set, and ε(H k →H k+1 ) represents a set of edges from H k to H k+1 ;

[0033]

[0034] 3) The fully connected edges are applied to the rooted tree, and an adjacency matrix with a larger receptive field and a far connectivity meaning is constructed The adjacency matrix As shown in formula (2), the constructed hierarchical edge set is as shown in formula (3), wherein H k ∪H k+1 represents a union set of the kth hierarchical node set and the kth hierarchical node set;

[0035]

[0036] ε k =ε(H k ∪H k+1 ,H k →H k+1 ,H k+1 →H k ) (3)

[0037] The joint feature input is in the form of three-dimensional coordinates, and the size is C*T*2N, wherein V is the number of human body joints, and N is the number of joints. Two action performers are taken as a whole to be input into the network, the video frame number is set to 120 frames, and the joint input data size is 3*300*50. The obtained hierarchical set adjacency matrix is normalized by using a degree matrix to ensure the stability of training, and all elements of the matrix are taken as learnable parameters to ensure the adaptability of training.

[0038] S2: The CTR-GC channel topology optimization graph convolution is constructed as a time decoupling graph convolution, as shown in Figure 2As shown, using the relationship matrix obtained in S1, a THGC graph convolution block is constructed to capture time channel information and spatial channel information at the same time;

[0039] 1) In order to obtain more spatio-temporal feature information, the node data is input, converted into a relationship matrix by S1, and refined into three parts. The first branch extracts spatial feature information through a spatial channel module; the second branch extracts temporal feature information through a time decoupling module; and the third branch is a branch using a 1*1 convolution to obtain the residual connection of its own features;

[0040] 2) The input feature of the first branch is C / r*T*2N, which is extracted through a 1*1 convolution and a time average pooling module to obtain spatial feature information, and the feature shape becomes C / r*2N. The obtained spatial features are adjusted in feature shape to C / r*2N*2N using a self-contrast subtraction method, the output value is adjusted to the interval of -1 to 1 using a tanh activation function, and the feature shape is adjusted to C'*2N*2N using a 1*1 convolution;

[0041] 3) The input feature of the second branch is C'*T*2N, which is extracted through a 1*1 convolution and a time decoupling module to obtain spatial feature information, and the feature shape becomes T*2N. The obtained spatial features are adjusted in feature shape to T*2N*2N using a self-contrast subtraction method, the output value range is adjusted using a tanh activation function, and the obtained features are fused with the features obtained by a 1*1 convolution to obtain temporal feature information;

[0042] 4) The features obtained by the first branch and the second branch are added element by element with the hierarchical relationship matrix constructed in step 1 to obtain spatio-temporal features, and the obtained spatio-temporal features are fused with the features obtained by a 1*1 convolution to obtain the output spatio-temporal features;

[0043] Compared with the original CTR-GC module, the improved new module uses a time decoupling method to obtain rich temporal features, uses different methods in the two branches to extract node features of time and space, and uses a residual network structure in the second branch. Secondly, the time decoupling convolution block adds each layer of features passing through the time channel and the spatial channel with the hierarchical relationship matrix element by element and sends it to the next layer for fusion, so that the final output contains the feature map containing time and space information.

[0044] S3: The graph convolution in THGC is constructed as BLOCKGC, and then as TBGC graph convolution block;

[0045] Specifically,

[0046] 1) The input data is processed by standardizing the weight, the relative position encoding is introduced into the graph convolution, the shortest distance method is used to process the skeleton graph G sThe relative distance between the two upper joints is encoded as shown in formula (4), where P1 and P P The first and last vertices on path P are represented by the weight parameter B, which can be retrieved from the training parameter table. i,j Then, based on the shortest path distance d through the bone connection... i,j Assign to each joint pair.

[0047]

[0048] 2) The acquired graph encoding is batch normalized and a 1×1 convolution is used as the residual connection of the downsampling layer. The result is then processed by the ReLU activation function and output.

[0049] Following the CTR-GCN architecture, such as Figure 4 As shown, stacking TB-GC graph convolutional blocks and multi-scale temporal convolutional blocks yields a temporally decoupled hierarchical graph convolutional network, ensuring that the network maintains high recognition speed while improving recognition accuracy. The multi-scale temporal convolutional blocks are shown below. Figure 2 As shown;

[0050] Step 5: The fused input data is fed into an improved temporally decoupled hierarchical graph convolutional network for training and testing. Deep spatiotemporal features exhibiting natural, non-natural, and interactive connections are extracted. These features are then processed through pooling layers, fully connected layers, and activation functions to obtain the final classification result. The final recognition accuracy is 97.9%, demonstrating the effectiveness of the proposed method. The confusion matrix obtained during testing is shown below. Figure 6 As shown.

[0051] Experiments have demonstrated that the proposed method for recognizing two-person interactive behaviors based on temporally decoupled hierarchical graph convolution can effectively address the problem of insufficient modeling for the temporal continuity of interactive behavior recognition, solve the problem of poor real-time performance in recognizing large amounts of interactive behavior, and effectively improve the accuracy of complex interactive scenarios, thus greatly enhancing the efficiency of two-person interactive behavior recognition.

[0052] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these changes and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A two-person interaction behavior recognition method based on time-decoupled hierarchical graph convolution, characterized in that, Comprise: S1: the optimized joint data as the input of the network, the hierarchical graph topology is constructed to obtain the hierarchical relationship matrix; S2: the CTR-GC channel topology optimization graph convolution is constructed as a time decoupling hierarchical graph convolution, the hierarchical relationship matrix obtained in S1 is used to construct a THGC graph convolution block capable of capturing time channel information and space channel information at the same time; S3: the graph convolution in THGC is constructed as BLOCKGC, and then as a TBGC graph convolution block; S4: the TBGC graph convolution block and the multi-scale time convolution block are stacked to obtain a TB-GCN graph convolution network; S5: the TB-GCN graph convolution network is used for deep feature extraction and classification of the input human skeleton data, and a dual-person interaction behavior recognition result is obtained; The optimized joint data in S1 is the joint data after removing noise and frame loss, and the optimization processing method is: the joint data of two persons is obtained through a depth camera Kinect v2, the obtained joint data is denoised and filtered for frame loss, three strategies of calculating the frame length, dispersion degree and motion value of the node data are used to denoise and filter the joint data of two persons, the abdominal node is taken as the center to construct a rooted tree, the human joint nodes are divided into five layers, and the graph is divided so that the nodes in the same hierarchical structure edge set are stored in the same semantic space; The THGC graph convolution block constructed in S2 is composed of a residual structure composed of two branches and a 1*1 convolution, one branch passes through a 1*1 convolution, a time pooling module to further extract more detailed spatial feature information, the obtained spatial features are processed by self-contrast subtraction and tanh activation function, and the channel number is adjusted by 1*1 convolution to obtain spatial features; The other branch is composed of a bottleneck block composed of two flow branches, one flow passes through a spatial pooling module to extract more detailed time feature information, uses self-contrast subtraction and tanh activation function to process the obtained time feature, and the other flow adjusts the channel number by 1*1 convolution to retain its original feature information, the two flow features are fused to obtain time features, the obtained time features are multiplied and added with the spatial features obtained in the previous branch and the weighted hierarchical relationship matrix to obtain space-time features, the features fused with time and space are fused with the features of the self-features after 1*1 convolution, and a time decoupling hierarchical graph convolution block is constructed; In S3, the graph convolution in THGC is constructed as BLOCKGC, and then as a TBGC graph convolution block, including: the input data is processed by standardizing the weight, the relative position coding is introduced into the graph convolution, the adjacency matrix is generated to generate k-hop graph distance coding and calculate the weight, the batch normalization is performed and the 1*1 convolution is used as the residual connection of the down-sampling layer, the result is output by the ReLU activation function.

Images