Multi-person interaction recognition method based on hierarchical multi-scale space-time diagram convolutional network

Abstract

The invention discloses a multi-person interaction recognition method based on a hierarchical multi-scale space-time diagram convolutional network, and aims to solve the core problems of single multi-scale time sequence modeling, unbalanced global-local context and rough two-person hierarchical interaction in multi-person interaction recognition in a complex scene. The method comprises the following steps: firstly, preprocessing input multi-person skeleton data, and depicting a single-human-body joint dependence and cross-human-body interaction relationship in parallel through double-adjacency graph convolution; a multi-scale time sequence convolution module (MST-CAF) is used for capturing diversified motion modes from instantaneous motion to long-range motion; then, the fine joint details and the overall scene context are dynamically fused through a global-local double-branch interaction enhancement module (GL-DSIA); and finally, explicitly modeling a multi-person high-order relationship through a hierarchical attention mechanism, and outputting an interaction behavior recognition result.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, biometric recognition and behavior analysis, specifically to a multi-person interaction recognition method based on hierarchical multi-scale spatiotemporal graph convolutional network (H-MST-GCN), which is applicable to scenarios such as intelligent monitoring, human-computer interaction, virtual reality, and elderly care that require real-time understanding of dynamic multi-person interaction behavior. Background Technology

[0002] In the field of computer vision, skeleton-based behavior recognition has become a core technology for multi-person interaction analysis due to its robustness to changes in lighting, occlusion, and viewpoint. With the maturity of depth sensors (such as Kinect) and pose estimation algorithms, skeleton data can now stably acquire the 3D coordinates of more than 25 key joints of the human body, providing high-quality input for multi-person interaction recognition.

[0003] In existing technologies, the Spatiotemporal Graph Convolutional Network (ST-GCN) achieves single-person action recognition through "fixed adjacency matrix + temporal convolution", but its fixed topology cannot adapt to the dynamically changing joint dependencies in multi-person interactions (such as the need to strengthen the hand-to-torso connection when "hugging", while the need for this connection when "standing in parallel"). To address the issue of multi-person interaction, STGCN-PAM-TPIR introduces Paired Adjacency Matrix (PAM) to explicitly construct cross-human joint connections. However, it still suffers from three major drawbacks: 1) Single temporal modeling: Using fixed 3×1 or 9×1 convolutional kernels, it cannot simultaneously cover the multi-scale rhythms of "instant high-five" (0.2 seconds) and "continuous hug" (2 seconds), resulting in the loss of key dynamic information; 2) Global-local imbalance: Only the features of two people are concatenated in the last layer, without integrating "global trunk posture" (such as trunk rotation when "handing over an object") and "local joint details" (such as fingertip distance), which can easily misjudge "failed handing over an object" as "waving"; 3) Coarse multi-person fusion: Undifferentiated average pooling is used for multi-person features, ignoring the feature asymmetry between "force exertor and recipient" (such as the active party extending their arm and the passive party retreating in "pushing"), which weakens the semantic distinguishability of the interaction.

[0004] The aforementioned shortcomings result in low recognition accuracy and poor generalization ability of existing methods in complex scenarios (such as crowded areas with many people and changing interaction rhythms), making it difficult to meet the real-time and reliability requirements of practical scenarios such as intelligent monitoring and elderly care. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a multi-person interaction recognition method based on a hierarchical multi-scale spatiotemporal graph convolutional network. Through the collaborative design of "dual adjacency graph convolution + multi-scale temporal modeling + global-local fusion + hierarchical attention", the core challenge of multi-person interaction recognition is solved.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] S1: Multi-person skeleton data preprocessing, length standardization of skeleton sequences: all sequences are resampled to T=64 frames through linear interpolation to ensure consistent input temporal length; abnormal samples are removed, removing samples with joint confidence below 0.8 or joint coordinates outside the reasonable range; the data is reshaped into B×C×T×V×M format (B is the batch size, C=3 corresponding to x / y / z channels) to provide standardized input for subsequent graph convolution modules.

[0008] S2: Dual-Adjacency Graph Convolution Feature Extraction: Construct a Dual-Adjacency Graph Convolution (Dual-GCN) module to capture "single-human joint motion" and "cross-human interaction relationships" in parallel, overcoming the limitation of fixed topology in traditional GCNs.

[0009] S2.1 Single Human Adjacency Matrix A: Based on human physiological structure and joint spatial distance, three subgraphs (physical adjacency, symmetric adjacency, and self-loops) are constructed. The joint dependency strength is dynamically adjusted through learnable edge weights α, and the single human feature F is calculated. A :

[0010]

[0011] Among them, A k For a single human body, the kth order adjacent basic nucleus, Let be a learnable weight matrix, (α) k ⊙A k )·X represents standardized skeleton data, where "⊙" represents element-wise product and "·" represents matrix multiplication;

[0012] S2.2 Cross-human adjacency matrix B: The PAM strategy is used to expand the V joints of M individuals into an M×V fully connected graph, and the interaction relationship of "who acts on whom" is explicitly encoded by the learnable edge weight β.

[0013]

[0014] Among them, B k For the k-th adjacent base nucleus across the human body, Let X be a learnable weight matrix. flat The skeleton data after dimensional reshaping;

[0015] S2.3 Feature Fusion: Combining single human body features F A With cross-human characteristics F B By directly adding them along the channel dimension, we obtain the graph convolutional feature F that fuses the two types of information. GCN It retains the dynamic movements of a single person while also injecting cross-human interaction signals.

[0016] S3. Multi-scale Temporal Convolution Modeling: A multi-scale temporal convolution module (MST-CAF) is designed to address the problem of fixed receptive fields in traditional temporal convolution, capturing motion patterns from instantaneous to long-term.

[0017] S3.1 Channel Compression and Branch Generation: F is compressed using a 1×1 convolution. GCN The number of channels is compressed from C to C / 4, generating 4 parallel branch features, reducing computational overhead;

[0018] S3.2 Multi-scale Convolution: Perform depthwise separable temporal convolution on each of the four branches. The kernel size k is 3, 5, 7, or 9 (where 3 frames → instantaneous action and 9 frames → long-range dependency). The padding is set to (k-1) / / 2 to ensure that the output temporal length is consistent with the input.

[0019] S3.3 Adaptive Weight Fusion: Attention weights ω are generated through global average pooling and Softmax, dynamically adjusting the contributions of the four branches, and finally obtaining multi-scale temporal features F through residual connections. ms .

[0020] S4. Global-Local Dual-Branch Fusion:

[0021] S4.1: F ms Local features X are evenly divided along the channel dimension. L With global feature X G ;

[0022] S4.2: Local branching generates a local attention map A through grouped 1D convolutions and depthwise separable 1D convolutions. L , with X L Element-wise product yields locally enhanced features F L :

[0023] A L =σ(DW-Conv1D(X) L ))

[0024] Where σ is the Sigmoid activation function and DW-Conv1D is a depthwise separable 1D convolution;

[0025] S4.3: The global branch generates a global gated vector A through global spatiotemporal average pooling and two layers of 1×1 convolution. G , with X G Element-wise product yields the global enhanced feature F G :

[0026] F G =A G ⊙X G

[0027] S4.4: Generate pixel-level gated tensor G (G=σ(Conv1×1(GAP(X))), GAP is global average pooling), and dynamically fuse F according to G. L With F G :

[0028] F out =G⊙F L +(1-G)⊙F G

[0029] F out With F ms Residual connections are used to obtain global-local fusion features;

[0030] S5. Hierarchical Attention Aggregation and Recognition:

[0031] S5.1 Grouping and Self-Attention: Dividing the global-local features of M individuals into N... g A non-overlapping subsequence (each group contains 2 individuals) is used to perform multi-head self-attention on each group to enhance intra-group interactions (such as the feature association between the active and passive parties in "pushing").

[0032] S5.2 Feature Concatenation and Classification: The attention outputs of all groups are concatenated along the channel dimension and input into a two-layer fully connected network (hidden layer dimension 256, activation function ReLU), which is then mapped to the interaction category space (such as "handshake", "hug", "push" etc.).

[0033] The predicted probabilities of each category are then obtained through the Softmax function, and the category with the highest probability is taken as the final recognition result. During the training phase, the "standard path" (global average pooling) and the "hierarchical path" are optimized at the same time, and the mean of the two outputs is used to improve robustness during inference.

[0034] The beneficial effects of this invention are as follows:

[0035] Based on the inherent characteristics of multi-person interaction behavior—namely, "multi-scale temporal dynamics, global-local semantic association, and hierarchical interaction"—and the essential differences between different types of interaction behavior (such as handshakes, pushing, and hugging), this invention designs a multi-person interaction recognition method based on a hierarchical multi-scale spatiotemporal graph convolutional network (H-MST-GCN). This method utilizes dual-adjacent graph convolution to parallel characterize "single-person joint dependence and cross-person interaction relationships," multi-scale temporal convolution to comprehensively capture "instantaneous to long-range motion patterns," and global-local bi-branch fusion and hierarchical attention to deeply mine "detailed semantics and higher-order interactions" to extract features that reflect the essential state differences of different multi-person interaction behaviors. This method overcomes the limitations of current mainstream multi-person interaction recognition technologies, such as "fixed topology graph convolution cannot adapt to dynamic interactions, single temporal convolution is difficult to cover multi-rhythmic interactions, and coarse feature fusion erases interaction semantics." By leveraging the synergistic effect of dual adjacency graph convolution, multi-scale temporal convolution (MST-CAF), global-local bi-branch interaction enhancement (GL-DSIA), and hierarchical attention mechanism, it demonstrates excellent generalization performance on the NTU RGB+D 120 dataset interaction subset and the Kinetics multi-person action dataset, and can achieve good recognition results in practical application scenarios. Attached Figure Description

[0036] Figure 1 This is a flowchart of the multi-person interaction recognition method based on hierarchical multi-scale spatiotemporal graph convolutional network of the present invention;

[0037] Figure 2 The training curves for our proposed method and two existing methods are shown on the NTU RGB+D 120 dataset.

[0038] Figure 3 The graphs show the training curves obtained by the proposed method and two existing methods on the Kinetics dataset. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and preferred embodiments. The objectives and effects of the present invention will become clearer as a result. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Figure 1 As shown, the multi-person interaction recognition method based on hierarchical multi-scale spatiotemporal graph convolutional network of the present invention consists of five main steps.

[0040] 1. Multi-person skeleton preprocessing: Generate corresponding skeleton sequences and reshape the data into B×C×T×V×M format. This embodiment uses the publicly available datasets NTU RGB+D 120 and Kinetics. Both datasets provide various interaction data. The NTU RGB+D 120 dataset is the core validation set for multi-person interactions, while the Kinetics dataset is a generalization validation set.

[0041] 2. Dual Adjacency Graph Convolution Feature Extraction: Features are extracted from the preprocessed skeleton sequence, constructing a dual-branch structure of "single-person adjacency matrix A + cross-person adjacency matrix B" to extract single-person dynamic and cross-person interaction features in parallel. This includes the following sub-steps:

[0042] 2.1: Binarize each gait contour map to obtain a binary gait contour map where the area inside the human body region is 1 and the rest of the region is 0; construct a single human body adjacency matrix A, and construct three subgraphs: physical adjacency, symmetric adjacency, and self-loop. Dynamically adjust the joint dependency strength through learnable edge weights α, and calculate the single human body feature F. A ;

[0043] 2.2: Scan the binary gait contour map obtained in the previous step row by row, count the number of pixels with a value of 1 in each row, and generate a row scan statistics vector; construct the cross-human adjacency matrix B, and use the PAM strategy to expand the V=25 joints of M=2 individuals into a fully connected graph with M×V=50 nodes, initialize the edge weights, and calculate the cross-human interaction features F. B ;

[0044] 2.3 The features obtained in the first two steps are directly added together in the channel dimension to obtain the fused feature F. GCN .

[0045] 3. Multi-scale temporal convolution modeling, using 1×1 convolution to model F... GCN The number of channels is compressed to generate four parallel branch features (reducing computational overhead). Depthwise separable temporal convolutions are then performed on each of the four branches, with kernel sizes k of 3, 5, 7, and 9 (corresponding to different temporal receptive fields). Then, F... GCN Perform global average pooling and softmax to generate attention weights ω, and then sum them by weights to obtain F. ms and with F GCN Perform residual connections (to avoid gradient vanishing), and finally use a 1×1 convolution to restore the number of channels from 1 to C=3, outputting multi-scale temporal features F. ms .

[0046] 4. Constructing the Gait Energy Transfer Matrix: Based on the row scan statistics vector generated in the previous step, construct the gait energy transfer matrix as the final gait feature. Each column of the gait energy transfer matrix characterizes the gait energy distribution characteristics at a specific moment during normal walking; each row of the gait energy transfer matrix characterizes the gait energy transfer characteristics of a pedestrian within a gait cycle.

[0047] The specific steps for global-local dual-branch fusion are as follows:

[0048] 4.1 F ms Local features X are evenly divided along the channel dimension.L (Focusing on joint micro-movements) and global feature X G (Focusing on the overall posture);

[0049] 4.2 Local Branch X L Local attention map A is generated by grouped 1D convolutions (number of groups = 2) and depthwise separable 1D convolutions (kernel = 3). L Thus, the local enhancement feature F is obtained. L

[0050] F L =A L ⊙X L

[0051] 4.3 Global Branch X G A global gated A is generated through global spatiotemporal average pooling and two layers of 1×1 convolutions (ReLU activation). G The global enhancement feature F is obtained. G

[0052] F G =A G ⊙X G

[0053] 4.4 Gated dynamic fusion generates pixel-level gated tensors, which are then used to fuse the local enhanced features F obtained in the first two steps. L Global Enhancement Feature F G Perform dynamic fusion and with F ms Perform residual connections and output global-local fused features;

[0054] 5. Hierarchical grouping and attention aggregation: Divide the features obtained in the previous step into N... g Each subsequence (each group contains 2 interacting individuals) is processed, and the intra-group attention is calculated for each group. The attention outputs of all groups are concatenated along the channel dimension and then input into a two-layer fully connected network. The predicted probability of each category is obtained through the Softmax function, and the average of the two outputs is taken as the final recognition result.

[0055] Furthermore, in order to compare the accuracy of different methods in recognizing multi-person interactive behaviors, we compare the method of this invention with two other methods.

[0056] The following section describes how to perform multi-person interaction behavior recognition using the method of this invention and two existing methods, based on the NTU RGB+D 120 dataset and the Kinetics dataset:

[0057] (1) STGCN-PAM-TPIR method (C.-L. Yang, A. Setyoko, H. Tampubolon and K.-L. Hua, "Pairwise Adjacency Matrix on Spatial Temporal Graph Convolution Network for Skeleton-Based Two-Person Interaction Recognition," 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates, 2020, pp.2166-2170, doi:10.1109 / ICIP40778.2020.9190680);

[0058] (2) ST-GCN method (Xiuhui Wang, Wei Qi Yan, "Human Gait Recognition Based on Frame-by-Frame Gait Energy Images and Convolutional Long Short Term Memory", International Journal of Neural Systems, 2019); YAN S, XIONG Y, LIN D. Spatialtemporal graph convolutional networks for skeleton-based action recognition [C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2018:7444-7452.

[0059] The training curves of the proposed method and the two methods mentioned above are plotted on the NTU RGB+D 120 dataset and the Kinetics dataset, respectively. Figure 2 and Figure 3 As shown in the figure, the NTU RGB+D 120 dataset is divided into training and validation sets based on subject ID (Cross-Subject, CS); the Kinetics dataset uses the officially defined training, validation, and test sets. It can be seen that the method of this invention achieves higher recognition rates on both publicly available interactive behavior datasets.

Claims

1. A multi-person interaction recognition method based on a hierarchical multi-scale spatio-temporal graph convolution network, characterized in that, The method comprises the following steps: S1: Multi-person skeleton data preprocessing: obtaining an input multi-person 3D skeleton sequence, the skeleton sequence containing V joint coordinates of M interactive individuals; resampling the skeleton sequence to a fixed frame number T through linear interpolation, adjusting the data dimension to BxCXTxVM (B is batch size, C is channel number) after removing abnormal samples, and obtaining standardized skeleton data; S2: Dual-adjacency graph convolution feature extraction: based on the standardized skeleton data obtained in S1, a dual-adjacency graph convolution (Dual-GCN) module is constructed, and single-body joint features and cross-body interaction features are extracted in parallel; S2.1: Construct single-person adjacency matrix Divide into physical adjacency subgraph, symmetric adjacency subgraph, self-loop subgraph, through learnable edge weight Dynamically adjust joint dependency strength, calculate single-person feature F A . S2.2: Constructing the cross-body adjacency matrix The pair-wise adjacency matrix (PAM) strategy is adopted to extend to a fully connected graph of M x V nodes, with learnable edge weights Strengthen the interaction hotspots, calculate the cross-body features F B . S2.3: F A with F B In the channel dimension, directly add to get the graph convolution feature F GCN fusion single human and cross human information S3: Multi-scale temporal convolution modeling: Based on F GCN , capture multi-granularity motion dynamics through multi-scale temporal convolution module (MST-CAF): S3.1: F is compressed to C / 4 by 1x1 convolution GCN The number of channels is compressed to C / 4, generating 4 parallel branch features X1~X4; S3.2: Depth separable temporal convolution is performed on the four branch features respectively, the convolution kernel size k is 3, 5, 7 and 9 respectively, the spatial dimension is kept unchanged, and branch outputs branch1~branch4 are obtained; S3.3: to F GCN Perform global average pooling and 1x1 convolution, generate attention weight by Softmax Weighted sum of 4 branch outputs by weight, restore channel number and F GCN Make residual connection to get multi-scale time sequence feature F ms ; S4: Global-local dual branch fusion: in F ms Embedding global-local dual branch interactive enhancement module (GL-DSIA) on the top, dynamically balancing details and context: S4.1 dual-branch feature extraction, F ms along the channel dimension, local branch (X L ) captures joint micro-motions (e.g., finger tip distance changes) through grouped 1D convolutions with depthwise separable convolutions, global branch (X G ) captures overall motion rhythm (e.g., trunk rotation direction) through global spatio-temporal pooling; S4.2 Gated dynamic fusion: generate a pixel-level gated tensor G, adaptively adjust the fusion ratio of local features F L and global features F G . For example, in the "delivery" action, G→0.7 (local fingertip details are dominant), in the "walking side by side" action, G→0.3 (global posture is dominant), and finally through the residual connection, avoid "global drowning local" or "local covering global". S5: Hierarchical attention aggregation and recognition: modeling multi-person high-order relationships through a hierarchical attention mechanism, and outputting a recognition result: S5.1: Divide the global-local features of M individuals into N g non-overlapping subsequences (2 individuals per group) along the number dimension, and perform multi-head self-attention for each group: where Q i = K i = V i is the i-th group of features, d h is the single-head attention dimension; S5.2: The attention outputs of all groups are spliced along the channel dimension, input into a two-layer fully connected network to map to a category space, and an interactive behavior prediction probability is obtained; S5.3: The final multi-person interaction recognition result is output through the Softmax function.

2. The method of claim 1, wherein, The multi-scale temporal convolution module uses parallel temporal convolution branches with convolution kernel sizes of 3, 5, 7 and 9, and generates attention weights through global average pooling and Softmax for feature fusion.

3. The method of claim 1, wherein, The global-local dual-branch interaction enhancement module extracts global statistical features and local joint features through a double-flow structure, and dynamically weights and fuses them through a learnable gating mechanism.

4. The method of claim 1, wherein, The hierarchical attention mechanism divides multi-person features into groups, uses multi-head self-attention for information interaction in each group, and finally outputs a classification result through splicing and a fully connected layer.

Images