An action recognition method based on pyramid segment dynamic graph construction and ellipsoid geometric constraint metric learning
By constructing a pyramid-shaped segmented dynamic graph and learning ellipsoidal geometric constraints, the shortcomings of graph structure modeling in skeleton action recognition are addressed. This approach enables the capture of high-order semantic associations of non-adjacent joints of the human skeleton and accurate representation of action features, thereby improving the accuracy and robustness of action recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF INFORMATION SCI & TECH
- Filing Date
- 2026-05-08
- Publication Date
- 2026-06-02
Smart Images
Figure CN122135440A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision image processing technology, and in particular to an action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning. Background Technology
[0002] To overcome the computational and environmental adaptability bottlenecks brought about by high-dimensional visual data, human skeleton-based action recognition technology has gradually become a research hotspot in academia and industry as an efficient alternative. Compared with traditional methods, skeleton data obtains the 2D / 3D keypoint coordinate sequence of the human body through depth sensors or pose estimation algorithms. This modality greatly filters out irrelevant background noise and lighting interference, not only reducing the data dimensionality by orders of magnitude but also significantly improving the model's anti-interference ability in complex scenes.
[0003] Existing methods can generally be divided into two categories: (1) Using recurrent neural networks (RNNs) or convolutional neural networks (CNNs) to process coordinate sequences, simply treating them as linear time inputs or two-dimensional pseudo-images. The fundamental flaw of this approach is that it ignores the inherent non-Euclidean topology of the human skeleton. The human skeleton is essentially a natural graph structure composed of joints (nodes) and bones (edges). RNNs have difficulty establishing joint spatial connections across physical distances, while the regular grid convolution of CNNs will destroy the natural topological dependencies between joints. This structural destruction in representation makes it difficult for the model to capture the most discriminative spatiotemporal features of the action, which seriously restricts the further improvement of recognition accuracy. (2) Exploring the combination of graph convolutional networks (GCNs) to aggregate locally physically connected joint features. These methods usually use fixed human skeleton topology or only construct graph structures based on binary relationships between nodes, which makes it difficult to fully describe the high-order semantic relationships between non-adjacent joints. At the same time, the isotropic distance metric in traditional Euclidean space is difficult to accurately characterize the non-uniform distribution characteristics of different joints in the latent feature space in complex actions, which limits the ability to construct dynamic graphs. Furthermore, long temporal skeleton sequences have different temporal scale features at different semantic levels. Fixed single-scale mapping methods cannot take into account both local action changes and global action patterns, thus affecting the accuracy and robustness of action recognition. Summary of the Invention
[0004] Purpose of the invention: The purpose of this invention is to provide an action recognition method based on pyramid-style segmented dynamic graph construction and ellipsoidal geometric constraint metric learning, which solves the shortcomings of existing skeleton action recognition methods in graph structure modeling.
[0005] Technical Solution: The action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning described in this invention includes the following steps:
[0006] Step S1: Obtain the human skeleton action sequence, embed high-dimensional features into the coordinates of the skeleton joints and add spatial position encoding to construct the initial spatiotemporal features;
[0007] Step S2: In the graph convolutional unit, a feature space with ellipsoidal geometric constraints is constructed through learnable orthogonal rotation transformation and anisotropic scaling transformation. The distance between key features is calculated in the transformation space, and a differentiable dynamic hypergraph association matrix is generated based on the continuous nearest neighbor relationship mapping to capture the high-order semantic association between non-adjacent key points.
[0008] Step S3: Using a pyramid-style temporal segmentation strategy, the temporal features are divided into segments of different granularities at different depth levels of the network. Local dynamic graphs are constructed for each segment, and a global dynamic graph is constructed for the entire sequence, realizing multi-scale temporal relationship modeling from fine-grained local actions to coarse-grained global patterns.
[0009] Step S4: Adaptively weightedly fuse the generated dynamic hypergraph association matrix, the generated local dynamic graph, and the global dynamic graph with the preset static human body topology prior graph to obtain the fused dynamic graph structure.
[0010] Step S5: Based on the fused dynamic graph structure, perform spatial graph convolution and multi-scale temporal convolution on the initial spatiotemporal features in sequence to extract the deeply fused spatiotemporal features;
[0011] Step S6: Globally pool the spatiotemporal features of deep fusion and map them to the target action category through a classifier to complete action recognition.
[0012] Furthermore, in step S2, the construction of the feature space with ellipsoidal geometric constraints is specifically achieved by generating an orthogonal rotation matrix through a parameterized antisymmetric matrix to rotate the feature space, and introducing a positively valued diagonal scaling matrix to scale each dimension, thereby realizing the transformation of the feature space from isotropic to anisotropic.
[0013] Furthermore, in step S2, generating a differentiable dynamic hypergraph association matrix specifically involves: after calculating the distance between nodes in the feature space constrained by ellipsoidal geometry, replacing the traditional hard threshold truncation operation with a continuous nearest neighbor mapping mechanism based on temperature parameter control, and smoothly converting the distance values into a continuous similarity relation matrix.
[0014] Furthermore, in step S3, the pyramid-style temporal segmentation strategy is as follows: in the shallow layers of the network, a finer-grained temporal segmentation is used to construct multiple local segment dynamic graphs to capture local action details; in the deeper layers of the network, the segmentation granularity is gradually reduced until the deepest layer adopts a single-segment global graph construction method to capture global action semantics.
[0015] Furthermore, the construction of local segment dynamic graphs involves: for the features within each time segment, using a temporal attention pooling mechanism based on convolution and normalized exponential functions, clustering the multi-frame information within the segment into a single segment-level representation, and then constructing a dynamic graph based on this representation.
[0016] Furthermore, in step S4, the adaptive weighted fusion is specifically as follows: learnable non-negative weight parameters are introduced into the static human topology prior graph, the local dynamic graph, and the global dynamic graph respectively, and the multiple graph structures are weighted and summed.
[0017] Furthermore, in step S5, when performing spatial graph convolution, a zero-copy expansion technique is used to extend and align the temporal dimension of the segment-level dynamic graph to the full temporal length without copying, and the spatial graph convolution operation of all segments is completed by single operator fusion.
[0018] The action recognition system based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning described in this invention includes:
[0019] Initial feature embedding module: used to map the coordinates of the input human skeleton joints to a high-dimensional space and inject spatial position codes, outputting initial spatiotemporal features;
[0020] Ellipsoidal Metric Learning and Dynamic Hypergraph Generation Module: This module performs learnable orthogonal rotations and anisotropic scaling on the feature space, constructs ellipsoidal geometric constraints, and generates a differentiable dynamic hypergraph correlation matrix in the transformed space.
[0021] Pyramid-style dynamic graph construction module: used to construct local segment dynamic graphs and global dynamic graphs at different depths and layers of the network with different temporal segmentation granularities;
[0022] Multi-graph adaptive fusion module: used to perform non-negative fusion of dynamic hypergraph association matrix, local dynamic graph, global dynamic graph and static human prior graph with learnable weights to generate the final fused graph structure;
[0023] Spatiotemporal feature extraction module: used to extract deep spatiotemporal semantics from the initial spatiotemporal features based on the fusion graph structure;
[0024] Action classification module: Used to perform global average pooling on the extracted deep spatiotemporal features and output action category prediction results through a fully connected layer.
[0025] An electronic device according to the present invention includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the methods described herein.
[0026] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described herein.
[0027] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:
[0028] This invention takes a skeleton sequence as input. First, it embeds the original joint coordinates into a high dimension and injects spatial position encoding to obtain initial spatiotemporal features with spatial awareness. Then, it introduces a learnable distance metric with ellipsoidal geometric constraints within each graph convolutional unit. Anisotropic feature transformation is achieved through antisymmetric matrix parameterized rotation transformation combined with diagonal scaling parameters, and a differentiable dynamic hypergraph is constructed based on a continuous soft nearest neighbor mechanism. Furthermore, a pyramid-style temporal segmentation strategy is adopted to generate local segment dynamic graphs and global dynamic graphs at different network levels using different segmentation granularities. These are then adaptively fused with a static human topology prior graph to achieve multi-scale spatiotemporal relationship modeling. Finally, end-to-end action recognition is completed through spatial graph convolution, multi-scale temporal convolution, and a global pooling classification head. This invention significantly improves the model's recognition accuracy and generalization ability. This technology can be widely applied in key areas such as human-computer interaction, intelligent monitoring, and autonomous driving. Specifically, it can provide solid technical support for gesture interaction in smart homes, quantitative assessment of medical rehabilitation, early warning of abnormal behavior in public places, and safety monitoring of driver status. Attached Figure Description
[0029] Figure 1 This is a flowchart of the present invention. Detailed Implementation
[0030] The technical solution of the present invention will be further described below with reference to the accompanying drawings.
[0031] like Figure 1 As shown, an embodiment of the present invention provides an action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning, including the following steps:
[0032] Step S1: Obtain skeleton action sequence samples and organize the input samples into a tensor form containing batch dimension, channel dimension, time dimension, keypoint dimension, and human instance dimension. First, a linear mapping layer is used to map the low-dimensional coordinate features of each keypoint to a high-dimensional embedding space. Then, learnable spatial location encoding parameters are added to the embedded keypoint features to enhance the ability to express the spatial identity of keypoints and the topological position of the human body. Subsequently, the feature distribution is normalized through a nonlinear activation function and batch normalization operation to obtain an initial spatiotemporal representation. Through this step, the representational ability of subsequent dynamic graph construction and feature propagation can be improved while preserving the skeleton structure information.
[0033] Because the raw skeleton data (typically containing the 3D coordinates x, y, z of the joints and confidence scores) has low dimensionality, direct graph convolution can easily lead to a representation bottleneck. The raw skeleton data (containing batch, channel, temporal, joint, and human instance dimensions) suffers from low dimensionality in its three-dimensional coordinate system, making direct graph convolution prone to representation bottlenecks.
[0034] First, through linear layers Input features Mapped to a high-dimensional space, then injected with learnable spatial location codes. (Where 1 represents the batch and time dimension, V represents the joint dimension, and Cout represents the channel dimension), the encoded features are finally obtained. The formula is expressed as:
[0035]
[0036] Then, the tensor is reshaped and BatchNorm1d is applied to stabilize the feature distribution, generating an initial representation that combines skeleton structure priors with node identity identifiers.
[0037] Step S2: For the features obtained in Step S1, the node features are first mapped to multiple subspace representations in the graph convolutional unit; then, an antisymmetric parameter matrix is constructed for each subspace, and an orthogonal rotation matrix is generated through matrix exponential mapping to ensure the geometric stability of the feature rotation transformation; simultaneously, a learnable diagonal scaling parameter is introduced to perform positive scaling on the rotated features in each dimension to form an anisotropic feature space under ellipsoidal geometric constraints; in this transformation space, the pairwise distances between nodes are calculated, and the distances are converted into a continuous soft nearest neighbor matrix through a temperature-scaled Softmax operation; then, combined with the hyperedge weights adaptively predicted by the node features, the soft nearest neighbor matrix is normalized to generate a dynamic hypergraph adjacency matrix. Through this step, the limitation of fixed human physical connections can be overcome, implicit collaborative relationships between distant joints can be adaptively captured, and end-to-end differentiability of the dynamic graph construction process can be guaranteed.
[0038] This invention utilizes the concept of Mahalanobis distance to capture implicit dependencies across physical connections, and in its implementation, transforms it into Euclidean distance after feature space rotation and scaling. First, an antisymmetric matrix is constructed using the mapping from Lie algebra to Lie groups. And generate absolutely orthogonal rotation matrices through matrix exponential mapping. :
[0039]
[0040]
[0041] in For learnable parameter matrix, This is the transpose of the matrix. Then, a diagonal scaling matrix is generated using the sigmoid function and coefficient scaling. Node features The output node features are obtained by performing anisotropic transformations that first rotate and then scale. :
[0042]
[0043] Calculate the Euclidean distance between nodes in this distorted space. To avoid the non-differentiable faults and isolated nodes caused by traditional Top-K truncation, Continuous k-NN (temperature-scaled Softmax) is used to convert distances into soft nearest neighbor similarities and generate a hypergraph association matrix. :
[0044]
[0045] in Temperature hyperparameters are used to control the smoothness of the distribution. Ultimately, this will... Normalize the generation graph adjacency tensor.
[0046] Step S3: For the input temporal features, the time dimension is uniformly divided according to a preset number of segments in each graph convolutional unit, and the boundaries of continuous time segments are obtained by combining a remainder compensation mechanism. Temporal attention pooling based on convolution and Softmax is applied to the features within each time segment to obtain segment-level aggregated representations. Then, local segment dynamic graphs are constructed based on each segment-level aggregated representation. At the same time, average pooling is performed along the time dimension of the entire time series to construct a global dynamic graph. A pyramid-style temporal segmentation strategy is adopted in the network depth direction, so that shallow graph convolutional units use a smaller number of segments, medium graph convolutional units use a medium number of segments, and deep graph convolutional units use a single-segment global graph construction method, thereby realizing multi-scale temporal relationship modeling from local to global and from fine-grained to coarse-grained.
[0047] To extract macroscopic temporal dynamic features and reduce computational cost, segmented temporal attention pooling is performed on the features along the time dimension before the input graph generator. The network consists of three deep stages, which segment the long sequence into... A fragment.
[0048] For the Characteristics of a time period ,use Convolutional prediction temporal attention weights The feature vector is then activated by Softmax and subsequently weighted and summed to obtain a compressed feature vector. :
[0049]
[0050]
[0051] The pooled features are input into a shared graph generator to generate dynamic graphs with different macroscopic temporal receptive fields, perfectly balancing the need for computational complexity and temporal dynamic modeling.
[0052] Step S4: In each graph convolutional unit, firstly, learnable edge importance weights are applied to the static human body topology prior graph and normalized to obtain a static baseline graph. Then, the static baseline graph, local segment dynamic graph, and global dynamic graph are adaptively fused using non-negative learnable weights to obtain a fused dynamic graph for the corresponding time segment. Next, the fused segment dynamic graph is expanded along the time dimension and concatenated into a complete temporal dynamic graph adjacency tensor, and graph convolution propagation is performed on the node features. After completing the spatial graph convolution, temporal dynamic features are further extracted through a multi-scale temporal convolution module, and residual connections are introduced to enhance the training stability of the deep network. Through this step, the inherent physical topology of the human body, the relationships between local action segments, and the semantic relationships between global actions can be utilized simultaneously to achieve joint modeling and deep alignment of spatiotemporal features.
[0053] Obtain data-driven fragment animations With global dynamic graph Then, it is combined with learnable edge weights. Static human body physical benchmark topology diagram Fusion is performed. Learnable scalar parameters are introduced. and And its nonnegativity is guaranteed by ReLU:
[0054]
[0055] When performing graph convolution, zero-copy expansion is used to divide the graph fragments. The time dimension is extended and aligned to the actual time sequence length without copying, completely eliminating memory fragmentation caused by For loop-based slicing operations. Subsequently, spatial graph convolution is completed by fusing large matrices using a single large matrix Einsum operator.
[0056]
[0057] in The output tensor of the spatial graph convolution. The adjacency tensor of the fused graph represents the fully aligned graph. This represents the transformed node input features.
[0058] Next, multi-scale temporal receptive fields are captured by multi-scale temporal convolution (MultiScale-TCN, which uses different dilation rates to extract multi-scale temporal receptive fields), and residual connections are used to prevent gradient vanishing.
[0059] Step S5: After the three-stage dual-branch network extraction in Step S4, the number of output channels gradually increases to 512 dimensions, resulting in a spatiotemporal feature tensor with high-order semantics. Global average pooling is then performed along the time dimension T and the spatial node dimension V to completely remove the spatiotemporal resolution differences.
[0060] After a three-stage spatiotemporal feature extraction process, progressing from shallow to deep, the number of feature channels gradually increases exponentially. At the end of the model, multidimensional global average pooling (GAP) is employed. Let the shape of the output tensor at this point be... Along the spatiotemporal fusion dimension (time) With nodes ) and human instance dimension ( Perform aggregate mean calculation:
[0061]
[0062] This operation completely eliminates the temporal length and spatial resolution differences of the input video, retaining only the high-dimensional data that is most discriminative of the action. Semantic features.
[0063] Step S6: Finally, redundant activated neurons are filtered out through the Dropout layer to prevent overfitting, and the pooled feature vectors are mapped to the target action category space using a fully connected layer to output the final action classification prediction distribution, thus completing the end-to-end network training and action recognition.
[0064] Features after pooling Dropout is applied to randomly deactivate some neurons to effectively prevent overfitting, and the neurons are linearly mapped to the space of the target action class number via fully connected layers (FC):
[0065]
[0066] Output classification It will be used as the final action prediction distribution in the loss calculation and will drive the entire skeleton recognition network to perform end-to-end optimization.
[0067] It achieves excellent performance on the two mainstream semantic segmentation datasets NTU RGB+D 60 and NTU RGB+D 120. The comparison results with other methods are shown in Table 1.
[0068] Table 1 Comparison of the present invention with other action recognition methods. 。
Claims
1. An action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning, characterized in that, Includes the following steps: Step S1: Obtain the human skeleton action sequence, embed high-dimensional features into the coordinates of the skeleton joints and add spatial position encoding to construct the initial spatiotemporal features; Step S2: In the graph convolutional unit, a feature space with ellipsoidal geometric constraints is constructed through learnable orthogonal rotation transformation and anisotropic scaling transformation. The distance between key features is calculated in the transformation space, and a differentiable dynamic hypergraph association matrix is generated based on the continuous nearest neighbor relationship mapping to capture the high-order semantic association between non-adjacent key points. Step S3: Using a pyramid-style temporal segmentation strategy, the temporal features are divided into segments of different granularities at different depth levels of the network. Local dynamic graphs are constructed for each segment, and a global dynamic graph is constructed for the entire sequence, realizing multi-scale temporal relationship modeling from fine-grained local actions to coarse-grained global patterns. Step S4: Adaptively weightedly fuse the generated dynamic hypergraph association matrix, the generated local dynamic graph, and the global dynamic graph with the preset static human body topology prior graph to obtain the fused dynamic graph structure. Step S5: Based on the fused dynamic graph structure, perform spatial graph convolution and multi-scale temporal convolution on the initial spatiotemporal features in sequence to extract the deeply fused spatiotemporal features; Step S6: Globally pool the spatiotemporal features of deep fusion and map them to the target action category through a classifier to complete action recognition.
2. The action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning according to claim 1, characterized in that, In step S2, the feature space of the ellipsoidal geometric constraint is constructed by generating an orthogonal rotation matrix through a parameterized antisymmetric matrix to rotate the feature space, and introducing a positively valued diagonal scaling matrix to scale each dimension, thereby realizing the transformation of the feature space from isotropic to anisotropic.
3. The action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning according to claim 1, characterized in that, In step S2, generating a differentiable dynamic hypergraph association matrix specifically involves: after calculating the distance between nodes in the feature space constrained by ellipsoidal geometry, replacing the traditional hard threshold truncation operation with a continuous nearest neighbor mapping mechanism based on temperature parameter control, and smoothly converting the distance values into a continuous similarity relation matrix.
4. The action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning according to claim 1, characterized in that, In step S3, the pyramid-style temporal segmentation strategy is as follows: in the shallow layers of the network, fine-grained temporal segmentation is used to construct multiple local segment dynamic graphs to capture local action details; in the deeper layers of the network, the segmentation granularity is gradually reduced until the deepest layer adopts a single-segment global graph construction method to capture global action semantics.
5. The action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning according to claim 4, characterized in that, The construction of local segment dynamic graphs involves: for the features within each time segment, using a temporal attention pooling mechanism based on convolution and normalized exponential functions, clustering the multi-frame information within the segment into a single segment-level representation, and then constructing a dynamic graph based on the segment-level representation.
6. The action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning according to claim 1, characterized in that, In step S4, the adaptive weighted fusion is performed as follows: learnable non-negative weight parameters are introduced into the static human topology prior graph, the local dynamic graph, and the global dynamic graph, respectively, and the multiple graph structures are weighted and summed.
7. The action recognition method based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning according to claim 1, characterized in that, In step S5, when performing spatial graph convolution, a zero-copy expansion technique is used to extend and align the temporal dimension of the segment-level dynamic graph to the full temporal length without copying, and the spatial graph convolution operation of all segments is completed by single operator fusion.
8. An action recognition system based on pyramid-shaped segmented dynamic graph construction and ellipsoidal geometric constraint metric learning, characterized in that, include: Initial feature embedding module: used to map the coordinates of the input human skeleton joints to a high-dimensional space and inject spatial position codes, outputting initial spatiotemporal features; Ellipsoidal Metric Learning and Dynamic Hypergraph Generation Module: This module performs learnable orthogonal rotations and anisotropic scaling on the feature space, constructs ellipsoidal geometric constraints, and generates a differentiable dynamic hypergraph correlation matrix in the transformed space. Pyramid-style dynamic graph construction module: used to construct local segment dynamic graphs and global dynamic graphs at different depths and layers of the network with different temporal segmentation granularities; Multi-graph adaptive fusion module: used to perform non-negative fusion of dynamic hypergraph association matrix, local dynamic graph, global dynamic graph and static human prior graph with learnable weights to generate the final fused graph structure; Spatiotemporal feature extraction module: used to extract deep spatiotemporal semantics from initial spatiotemporal features based on the fusion graph structure; Action classification module: Used to perform global average pooling on the extracted deep spatiotemporal features and output action category prediction results through a fully connected layer.
9. An electronic device comprising a processor and a memory, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.