Skeletal motion recognition method based on spatiotemporal topology learning

Abstract

This invention relates to a skeletal action recognition method based on spatiotemporal topology learning, belonging to the field of computer vision and artificial intelligence technology. The method includes: constructing a two-dimensional topology coding module to simultaneously capture the static and dynamic topological relationships of the skeletal structure; employing a multi-level spatial attention aggregation module to adaptively weight and aggregate skeletal features at different levels to enhance the expressive power of key nodes and edges; and designing a temporal dynamic sequence pooling method to extract high-order dynamic information from time-series data based on a dynamic modality decomposition strategy, thereby improving the accuracy of action recognition. The advantages are: this method can fully utilize the spatiotemporal structural information of the skeleton data to achieve accurate recognition of human actions, while improving the modeling ability for complex dynamic behaviors, avoiding the limitations of traditional methods in capturing high-order dynamic information.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a skeletal action recognition method based on spatiotemporal topology learning. The method aims to improve the ability to model the spatiotemporal features of human skeletal data and provide an end-to-end deep learning framework to achieve accurate recognition of complex actions. Background Technology

[0002] The human skeleton is a dynamic structure composed of multiple joint nodes and their connections. Traditional modeling methods typically rely on simple joint coordinate sequences, neglecting the complex topological features of the skeletal structure. Despite significant progress in skeletal motion recognition technology, existing methods still face numerous challenges. First, current skeletal motion recognition methods fail to fully utilize the topological information of skeletal data. For example, actions such as jumping and waving involve not only the movement of individual joints but also the collaborative relationships between multiple joints, making it difficult for existing methods to accurately model these dependencies. Second, existing methods typically employ fixed skeletal topologies, making it difficult to adapt to variations in individual skeletons, resulting in insufficient generalization ability. Furthermore, while some methods introduce learnable topologies, this may lose the physical constraints of the original skeleton, leading to a lack of stability in the model's learning of topological relationships. Third, there are still shortcomings in modeling temporal information. Current mainstream methods include strategies based on recurrent neural networks (RNNs), graph convolutional networks (GCNs), and temporal convolutional networks (TCNs). These methods can capture temporal features to some extent, but most can only handle low-order temporal information and cannot effectively model the high-order dynamic features of complex actions. For example, in movements such as running and spinning, the temporal changes of the movements are not simply the sum of joint movements, but have complex temporal dependencies. Traditional methods mainly rely on temporal pooling (such as max pooling or average pooling) to aggregate global temporal features, which makes it difficult to extract true high-order temporal dynamic information, thus limiting the ability to recognize complex movements. Therefore, there is an urgent need for a model framework that can simultaneously solve key problems such as spatial topology modeling of skeletal data, temporal dynamic feature extraction, and adaptive topology learning, while meeting the requirements of efficient computation, accurately modeling complex movement relationships, and improving generalization ability in different scenarios.

[0003] In summary, to achieve accurate recognition of complex human movements, there is an urgent need for innovation in spatial topology modeling of skeletal data, temporal dynamic feature extraction, and adaptive topology learning. Summary of the Invention

[0004] The purpose of this invention is to provide a skeletal action recognition method based on spatiotemporal topology learning, which solves the technical problems of existing technologies that fail to fully utilize the topological structure of skeletal data, struggle to accurately model the dynamic relationships between joints, and lack the ability to model high-order temporal information. This invention systematically addresses the limitations of existing methods in topology modeling, temporal modeling, and feature extraction by constructing a two-dimensional topology encoding module, a multi-level spatial attention aggregation module, and a temporal dynamic sequence pooling method, significantly improving the accuracy and generalization ability of action recognition. This invention successfully improves the accuracy of human action recognition and is highly applicable to fields such as intelligent monitoring, autonomous driving, virtual reality, human-computer interaction, and medical rehabilitation, providing an efficient, accurate, and robust solution for complex action recognition tasks.

[0005] The main idea of ​​the skeletal action recognition method based on spatiotemporal topology learning in this invention is as follows: For the input human skeletal data sequence, firstly, a two-dimensional topology encoding module is used to jointly model the skeletal structure using static and dynamic topology. Static spatial relationship encoding preserves the inherent structural characteristics of the human skeleton, while dynamic spatial relationship encoding extracts temporal topological changes during joint movement, thereby enhancing the structural expressiveness of the skeletal data. Simultaneously, a multi-level spatial attention aggregation module is introduced to adaptively weight and aggregate skeletal features at different levels, enabling the model to effectively focus on key joints and important movement patterns, improving the accuracy of feature representation. Furthermore, this invention employs a temporal dynamic sequence pooling method, based on dynamic mode decomposition theory, to extract higher-order dynamic information from the time series more efficiently, overcoming the shortcomings of traditional temporal pooling methods in temporal dependency modeling. Ultimately, accurate action recognition is achieved on skeletal data, while possessing stronger spatiotemporal feature capture capabilities.

[0006] The above-mentioned objective of the present invention is achieved through the following technical solution:

[0007] The skeletal action recognition method based on spatiotemporal topology learning acquires the skeletal data to be processed, inputs it into a pre-trained skeletal action classification and recognition network model, and obtains the action classification result of the skeletal data to be processed; it includes the following steps:

[0008] S1. Skeletal data preprocessing: The input three-dimensional skeletal joint sequence data is preprocessed using methods including denoising, standardization and normalization to ensure the stability and consistency of the skeletal data.

[0009] S2, First stage of the skeletal motion recognition model: Extract the spatiotemporal topological features of the skeletal sequence, construct a two-dimensional topological coding module, construct static spatial relationship coding by using shortest path graph weights, construct dynamic spatial relationship coding by using graph sequences based on persistent barcodes, and analyze the static relationship of the skeletal structure and the dynamic topological changes during the motion process;

[0010] S3, the second stage of the skeletal motion recognition model: a multi-level spatial attention aggregation module is constructed based on hierarchical skeleton features. Through representative feature extractors and edge convolution operations, the feature weights of key motion parts are learned at multiple structural scales to achieve multi-scale structural saliency enhancement.

[0011] S4. The third stage of the skeletal motion recognition model: Design a time dynamic sequence pooling method, perform high-order modeling of temporal feature embedding vectors through Koopman operator theory and dynamic pattern decomposition, construct a dynamic matrix to capture the time dynamics in nonlinear motion sequences, and combine it with the SoftMax classifier for category recognition.

[0012] The skeletal data preprocessing described in step S1 specifically includes:

[0013] The input skeletal sequence data is normalized by scaling the coordinate values ​​of each joint within a preset range to eliminate scale differences between different data sources. Simultaneously, noise reduction is performed to remove noise data caused by sensor errors or environmental interference, improving data quality and stability. Furthermore, time alignment is applied to the skeletal sequence data to ensure that different action sequences have consistent start and end points in the time dimension, facilitating subsequent feature extraction and analysis.

[0014] Step S2 involves extracting the spatiotemporal topological features of the skeletal sequence and constructing a two-dimensional topological coding module. The specific method is as follows:

[0015] S2.1 Constructing a Static Spatial Encoding Component: A static spatial encoding component is designed to analyze the static relationships of skeletal structures. By utilizing graph distance metrics to quantify the relative positions of joint pairs in the skeletal diagram, an anatomical skeletal connection model is clearly established. First, the relative positions of joint pairs in the skeletal diagram are established using the following formula:

[0016]

[0017] Where M∈Path(G s ) is the skeletal map G s From vertex v i To vertex v j The set of all paths, |E| is the number of edges in path E, d ij From v i to v j The shortest path distance, M1 and M |E| These represent the first and last vertices of path M, respectively. Used to calculate the shortest path distance d i,j Convert to weight D ijDefine a weight parameter D for each pair of joints in the skeletal diagram. ij By finding the shortest path distance d between joint pairs ij The weights are mapped to the weight parameters, and a weight is assigned to each pair of joints. The weights are then used to update the adjacency matrix in the skeleton map to preserve the connection information between bones. Finally, the statically encoded topological code is added to the hidden features of the GCN to capture the basic structural information of the skeleton.

[0018] S2.2 Constructing a Dynamic Spatial Encoding Component: Design a dynamic spatial encoding component to analyze the dynamic topological relationships of the skeletal structure; given an input pose sequence, construct a weighted dynamic graph D. G Using bone joints as nodes and the Euclidean distance between joint pairs as edge weights, denoted as ω. ij According to ω ij If the threshold ∈ is exceeded, retain or delete the edge in the graph; the initial value of ∈ is set to 1; use persistent topological analysis to analyze the dynamic graph D. G The main idea behind filtering is to consider the filtering process. Instead of D G Treating it as a single object, the formula is as follows:

[0019]

[0020] This formula represents the process of starting from an empty set and gradually adding simplexes until a complete complex simplex S is constructed; where... i = 1, 2, ..., m denotes the abstract simplex complex number of each graph, where each S i Both can be considered as D G The subgraph or subcomplex is constructed by adding a new simplex at each step based on the previous simplex and newly introduced vertices or edges; simplexes are constructed from the graph by employing Vietoris-Rips complexes, determining the existence of edges based on the distance between vertices; the generation and disappearance of topological features (such as connected components) are captured; these dynamic topologies are encoded into persistent barcodes, summarizing the temporal appearance and disappearance of structural patterns; and the application of the differentiable vectorization function Ψ is used to further analyze these patterns. 0 These barcodes are converted into fixed-length vectors; then subjected to a linear projection f. θ A set of dynamic topological features K is generated and embedded into the feature space of the graph convolutional network. The specific formula is as follows:

[0021]

[0022] Among them, f θ It is a linear mapping layer defined by parameter θ, which vectorizes the function Ψ 0 The output is projected onto the feature space of the hidden layers of the graph convolutional network; Ψ0 Responsible for persistent barcodes Convert to a fixed-length feature vector; Representing 0-dimensional homology persistence barcodes, it encodes the persistence of connected components in the graph; finally, dynamic topological features K are embedded into the feature space of the graph convolutional network to provide the model with additional information about topological changes for specific actions.

[0023] In step S3, the weighted aggregation method using a multi-level spatial attention aggregation module is as follows:

[0024] S3.1 Representative Feature Extractor: The skeletal data calculated in step S1 and the static and dynamic drag structures calculated in step S2 are used to identify key representative nodes in each level using a representative feature extractor. The node features of each level are aggregated to obtain the representative feature set for each level, as shown in the following formula:

[0025]

[0026] Where Ψ represents the representative feature extraction operation, firstly extracting the features of representative nodes in each level, and then performing spatial average pooling on these features. F (m) T represents the feature at the m-th level. m K represents the set of nodes at level m. m The total number of nodes in T m+1 K represents the set of nodes at level m+1. m+1 The total number of nodes in K m Let K represent the set of nodes at level m. m+1 This represents the set of nodes at level m+1. This means maximizing the feature of node v along the time dimension t, i.e., selecting the maximum feature value of that node across all time steps. Then, for K... m and K m+1 Sum all nodes in T and divide by T. m +T m+1 ;

[0027] S3.2 Edge Convolution: After the representative feature extraction layer, since the features at each level do not yet share information, these feature maps need to be further integrated. All features at all levels are treated as nodes in a graph, and the similarity of nodes in the hierarchical feature space is learned based on the Euclidean distance between nodes. The final attention is the result of an edge convolution operation activated by the sigmoid function, expressed in the following formula:

[0028]

[0029] The function Ψ represents the feature F applied to each level. (m)The operation, where operator N represents the edge convolution operation applied at each level, and... This indicates the merging of features across all levels. The symbol ‖ indicates that the edge convolution operation is performed on the merged features. The sigmoid function σ is used to map the features to a range between 0 and 1, thereby representing the importance of each level of features.

[0030] Finally, the attention map is weighted and added to the feature map obtained from the convolution output to generate the final output feature map.

[0031] The temporal dynamic sequence pooling method designed in step S4 to extract higher-order dynamic information in the time dimension is as follows:

[0032] The Koopman operator enables the analysis of nonlinear systems through linear transformations in high-dimensional space; dynamic mode decomposition is used to approximate the Koopman operator in a computationally efficient manner; firstly, each eigenvector x is embedded using the embedding function ψ. t Embedded into a higher-dimensional space, the transitions between states are approximated by linear equations, as shown in the following formula:

[0033] ψx t+1 =(Kψ)(x t )

[0034] Where K denotes the Koopman operator. A linear approximation matrix D is constructed using dynamic mode decomposition, which controls the temporal dynamics of the sequence; given the characteristic matrix X with continuous time steps. 1:m-1 and X 2:m The formula for the transition matrix inspired by the Koopman operator is:

[0035]

[0036] Among them, X 1:m-1 X is the eigenma matrix with time steps from 1 to m-1. 2:m It is the feature matrix from time step 2 to m. This represents finding the pseudo-inverse matrix; matrix D yields the basic dynamic behavior of the sequence and is used as a pooling representation of the temporal structure; for classification, each action category is assigned a learnable matrix D that encodes its unique temporal features, and for a given sequence, D is used... i ·X t Calculate the evolutionary state and compare it with the next state X using the Euclidean distance. q For comparison, the formula is as follows:

[0037]

[0038] s i =-d i

[0039] s i It is the inverse of the distance, which is regarded as the classification score. SoftMax is then used to make the final prediction of all class scores:

[0040]

[0041] Where p i is the probability of the i-th class, and N is the total number of classes.

[0042] The beneficial effects of this invention are as follows:

[0043] 1. The skeletal action recognition method based on spatiotemporal topology learning proposed in this invention overcomes the limitations of existing methods by innovatively constructing a two-dimensional topology coding module. This module combines static and dynamic spatial relationship coding to fully exploit the static topological structure and dynamic motion patterns of skeletal data. Compared to traditional graph convolutional network methods, this invention can more accurately model skeletal topological relationships, avoiding the loss of topological information, thereby improving the accuracy and stability of action recognition.

[0044] 2. A multi-level spatial attention aggregation module is employed to effectively enhance the extraction capability of key skeletal features. Through a multi-scale attention mechanism, this invention can aggregate skeletal node features at different levels, highlighting key action parts and improving the modeling capability of human skeletal dynamic information. Compared with traditional pooling or averaging aggregation methods, the multi-level spatial attention aggregation module can adaptively adjust the feature weights of different parts, making the model more robust in complex action recognition tasks.

[0045] 3. This invention proposes a time-dynamic sequence pooling method to capture high-order dynamic information of time-series data, thereby improving time modeling capabilities. Utilizing Koopman theory, this invention constructs a dynamic matrix of time-series features, overcoming the limitation of traditional time pooling methods that can only capture first-order time information. This enhances the ability to recognize actions spanning long periods, making the system more advantageous in processing continuous actions (such as dance and sports). Compared to traditional graph convolutional network structures, this invention reduces redundant computation and improves computational efficiency, maintaining high-efficiency inference capabilities even on large-scale datasets.

[0046] 4. Applicable to various application scenarios, possessing excellent generalization ability and practical value. The method of this invention not only achieves excellent recognition performance on publicly available action recognition datasets (NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA), but also can be applied to multiple fields such as intelligent security, behavior analysis, human-computer interaction, and medical rehabilitation. Its good generalization ability allows the method to be extended to different types of skeletal data, improving the applicability and flexibility of practical applications. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate the invention and are used to explain it, but do not constitute an undue limitation of the invention.

[0048] Figure 1 This is a flowchart illustrating the process framework for establishing this invention;

[0049] Figure 2 This is a diagram illustrating the steps involved in establishing the present invention;

[0050] Figure 3 This is a diagram of the skeletal motion recognition network structure based on spatiotemporal topology learning according to the present invention.

[0051] Figure 4 This is a structural diagram of the two-dimensional topology coding component of the present invention;

[0052] Figure 5 This is a structural diagram of the multi-scale attention aggregation component of the present invention;

[0053] Figure 6 This is a diagram of the time-dynamic sequence pooling structure of the present invention;

[0054] Figure 7 This is a visualization of the t-SNE results on the NTU RGB+D 60 dataset, representing a specific embodiment of the present invention. Detailed Implementation

[0055] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] See Figures 1 to 7 As shown, the skeletal motion recognition method based on spatiotemporal topology learning of this invention first preprocesses the raw data, including data normalization and coordinate adjustment. The training set and validation set are divided in an 8:2 ratio. Training is performed using the training set, and the error between the actual output result and the label value is calculated. The difference is propagated from top to bottom using the backpropagation algorithm, and the weights are updated using the weights. After training, the trained neural network model is saved, and the parameters are adjusted using the validation set to perform a preliminary evaluation of the network model's training performance. Specifically, the method includes the following steps:

[0057] S1. Skeletal Data Preprocessing: The input skeletal sequence data is normalized by scaling the coordinate values ​​of each joint within a preset range to eliminate scale differences between different data sources. Simultaneously, noise reduction processing is performed to remove noise data caused by sensor errors or environmental interference, improving data quality and stability. Furthermore, time alignment is applied to the skeletal sequence data to ensure that different action sequences have consistent start and end points in the time dimension, facilitating subsequent feature extraction and analysis.

[0058] S2, First stage of the skeletal motion recognition model: Extract the spatiotemporal topological features of the skeletal sequence, construct a two-dimensional topological coding module, construct static spatial relationship coding by using shortest path graph weights, construct dynamic spatial relationship coding by using graph sequences based on persistent barcodes, and analyze the static relationship of the skeletal structure and the dynamic topological changes during the motion process;

[0059] S2.1 Design of static spatial coding components:

[0060] The static spatial coding component explicitly establishes an anatomical skeletal connection model by utilizing graph distance metrics to quantify the relative positions between joint pairs in a skeletal graph. The static coding formula is shown below:

[0061]

[0062] Where M∈Path(G s ) is the skeletal map G s From vertex v i To vertex v j The set of all paths, |E| is the number of edges in path E, d ij From v i to v j The shortest path distance, M1 and M |E| These represent the first and last vertices of path M, respectively. Used to calculate the shortest path distance d i,j Convert to weight D ij Define a weight parameter D for each pair of joints in the skeletal diagram. ij By finding the shortest path distance d between joint pairs ij This is mapped to weight parameters, assigning a weight to each pair of joints. This weight is then used to update the adjacency matrix in the skeleton graph to preserve bone connection information.

[0063] S2.2 Design of dynamic spatial coding components:

[0064] Construct a weighted dynamic graph D from a given input pose sequence. GIn this system, skeletal joints are treated as nodes, and the Euclidean distance between joint pairs is used as the edge weight ω. ij According to ω ij If the threshold ∈ is exceeded, retain or delete the edge in the graph. The initial value of ∈ is set to 1. Persistent topology analysis is used to analyze the dynamic graph D. G The main idea behind filtering is to consider the filtering process. Instead of D G Treating it as a single object, the formula is as follows:

[0065]

[0066] The formula represents the process of starting from an empty set and gradually adding simplexes until a complete complex simplex S is constructed. i = 1, 2, ..., m denotes the abstract simplex complex number of each graph, where each S i Both can be considered as D G The subgraph or subcomplex is constructed by adding a new simplex at each step based on the previous simplex and newly introduced vertices or edges; simplexes are constructed from the graph by employing Vietoris-Rips complexes, determining the existence of edges based on the distance between vertices; the generation and disappearance of topological features (such as connected components) are captured. These dynamic topologies are encoded into persistent barcodes, summarizing the temporal appearance and disappearance of structural patterns; and the application of the differentiable vectorization function Ψ is used to... 0 These barcodes are converted into fixed-length vectors; then subjected to a linear projection f. θ A set of dynamic topological features K is generated and embedded into the feature space of the graph convolutional network. The specific formula is as follows:

[0067]

[0068] Among them, f θ It is a linear mapping layer defined by parameter θ, which vectorizes the function Ψ 0 The output is projected onto the feature space of the hidden layers of the graph convolutional network; Ψ 0 Responsible for persistent barcodes Convert to a fixed-length feature vector; Representing 0-dimensional homology persistence barcodes, it encodes the persistence of connected components in the graph; finally, dynamic topological features K are embedded into the feature space of the graph convolutional network to provide the model with additional information about topological changes for specific actions.

[0069] S3. A multi-level spatial attention aggregation module is adopted to perform weighted aggregation of skeletal node features at different levels, enhance key skeletal edge information, and optimize feature expression capabilities.

[0070] S3.1 First, use a representative feature extractor to identify key representative nodes in each level:

[0071] The skeletal data computed in process S1 and the static and dynamic drag structures computed in process S2 are used to identify key representative nodes in each level using a representative feature extractor. The node features of each level are then aggregated to obtain the representative feature set for each level, as shown in the following formula:

[0072]

[0073] Ψ, where Ψ represents the representative feature extraction operation. First, features of representative nodes in each level are extracted, and then spatial average pooling is performed on these features. F (m) T represents the feature at the m-th level. m K represents the set of nodes at level m. m The total number of nodes in T m+1 K represents the set of nodes at level m+1. m+1 The total number of nodes in K m Let K represent the set of nodes at level m. m+1 This represents the set of nodes at level m+1. This means maximizing the feature of node v along the time dimension t, i.e., selecting the maximum feature value of that node across all time steps. Then, for K... m and K m+1 Sum all nodes in T and divide by T. m +T m+1 ;

[0074] S3.2, Using edge convolution to learn the similarity of nodes in the hierarchical feature space:

[0075] After the representative feature extraction layer, since the features at each level do not yet share information, these feature maps need to be further integrated. All features at all levels are treated as nodes in a graph, and the similarity of nodes in the hierarchical feature space is learned based on the Euclidean distance between nodes. The final attention is the result of an edge convolution operation activated by the sigmoid function, as expressed in the following formula:

[0076]

[0077] The function Ψ represents the feature F applied to each level. (m) The operation, where operator N represents the edge convolution operation applied at each level, and... This indicates the merging of features across all levels. The symbol ‖ indicates that the edge convolution operation is performed on the merged features. The sigmoid function σ is used to map the features to a range between 0 and 1, thereby representing the importance of each level of features.

[0078] Finally, the attention map is weighted and added to the feature map obtained from the convolution output to generate the final output feature map.

[0079] S4. Design a time-dynamic sequence pooling method to extract high-order dynamic information in the time dimension, and finally complete the recognition of the entire skeletal behavior.

[0080] The Koopman operator enables the analysis of nonlinear systems through linear transformations in high-dimensional space; dynamic mode decomposition is used to approximate the Koopman operator in a computationally efficient manner; firstly, each eigenvector x is embedded using the embedding function ψ. t Embedded into a higher-dimensional space, the transitions between states are approximated by linear equations, as shown in the following formula:

[0081] ψx t+1 =(Kψ)(x t )

[0082] Where K denotes the Koopman operator. A linear approximation matrix D is constructed using dynamic mode decomposition, which controls the temporal dynamics of the sequence; given the characteristic matrix X with continuous time steps. 1:m-1 and X 2:m The formula for the transition matrix inspired by the Koopman operator is:

[0083]

[0084] Among them, X 1:m-1 X is the eigenma matrix with time steps from 1 to m-1. 2:m It is the feature matrix from time step 2 to m. This represents finding the pseudo-inverse matrix; matrix D yields the basic dynamic behavior of the sequence and is used as a pooling representation of the temporal structure; for classification, each action category is assigned a learnable matrix D that encodes its unique temporal features, and for a given sequence, D is used... i ·X t Calculate the evolutionary state and compare it with the next state X using the Euclidean distance. q For comparison, the formula is as follows:

[0085]

[0086] s i =-d i

[0087] s i It is the inverse of the distance, which is regarded as the classification score. SoftMax is then used to make the final prediction of all class scores:

[0088]

[0089] Where p i is the probability of the i-th class, and N is the total number of classes.

[0090] S5. Define the network model loss function, and train and validate the network structure of this invention on the training and validation sets through multiple iterations. Use cross-entropy loss. To punish classification errors.

[0091] The network structure proposed in this invention is named TE-STGCN. To fully evaluate the performance of TE-STGCN in skeletal motion recognition tasks, three mainstream large-scale skeletal motion recognition datasets—NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA—were selected as benchmarks. Experiments were conducted under cross-subject (X-Sub), cross-view (X-View), and cross-set (X-Set) protocols, and the results were compared with current state-of-the-art methods to verify the effectiveness and superiority of this invention.

[0092] 1. Introduction to the dataset

[0093] The NTU RGB+D 60 dataset is a large-scale skeletal motion recognition dataset consisting of 60 actions performed by 40 subjects, totaling 56,880 skeletal sequences. Data was collected from multi-view RGB-D sensors, covering three main categories: daily activities, health-related actions, and social interactions. The dataset includes 3D skeletal joint coordinate information, with each skeleton consisting of 25 joints. During the experiments, the official dataset partitioning method was strictly followed, employing both cross-subject (X-Sub) and cross-view (X-View) evaluation protocols. In the X-Sub protocol, the 40 subjects were randomly assigned to training and testing sets, while in the X-View protocol, partitioning was based on camera angle.

[0094] The NTU RGB+D 120 dataset is an extended version of NTU RGB+D 60, adding 60 new action classes, for a total of 120 classes and containing 114,480 skeletal action sequences. This dataset was tested under cross-subject (X-Sub) and cross-set (X-Set) evaluation protocols, with the X-Set protocol using a partitioning of even-numbered ID classes for both training and testing.

[0095] 2. Evaluation Indicators

[0096] To quantitatively analyze the performance of the proposed TE-STGCN network, Top-1 accuracy was used as the evaluation metric for action recognition, and the calculation formula is as follows:

[0097]

[0098] Wherein, TP (True Positive) represents correctly classified action samples, FP (False Positive) represents misclassified action samples, and FN (False Negative) represents incorrectly identified action samples.

[0099] Furthermore, this invention uses t-SNE visualization to perform visual analysis of the model's classification ability, observe the model's ability to distinguish between similar action categories, and calculate the misclassification rate between categories.

[0100] 3. Experimental Results

[0101] NTU RGB+D 60 dataset: As shown in Table 1, the TE-STGCN network of this invention achieves a Top-1 recognition accuracy of 92.3% under the X-Sub scheme and 96.7% under the X-View scheme, which are 1.1% and 1.4% higher than CTR-GCN, respectively. Furthermore, the method of this invention achieves superior recognition performance in 48 of the 60 action categories, particularly demonstrating better classification results for actions with large amplitude of motion (such as "jumping" and "waving") and subtle movements (such as "writing" and "drinking water").

[0102] Table 1: Quantitative Results of the NTU RGB+D 60 Dataset

[0103]

[0104]

[0105] NTU RGB+D 120 dataset: As shown in Table 2, TE-STGCN achieves a Top-1 recognition accuracy of 88.3% under the X-Sub scheme and 89.5% under the X-Set scheme, which are 1.4% and 1.2% higher than CTR-GCN, respectively.

[0106] Table 2: Quantitative Results of the NTU RGB+D 120 Dataset

[0107]

[0108]

[0109] See Figure 7The t-SNE visualization results of the feature space distribution of representative action classes under the X-View protocol on the NTU RGB+D 60 dataset are shown. Four different action categories were randomly selected for comparative analysis: "combing hair" (blue), "descending" (orange), "sitting down" (green), and "putting on shoes" (red). This invention generates compact, well-separated feature clusters, demonstrating enhanced action discrimination capabilities. Intra-class feature consistency is improved, ensuring that instances of the same action remain close together in the feature space. This improved geometric arrangement in the feature space indicates an enhanced ability of the model to distinguish fine-grained motion patterns. These visualization results confirm that the model of this invention generates compact, well-separated feature clusters, ensuring high intra-class consistency and inter-class discriminability.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made to the present invention should be included within the scope of protection of the present invention.

Claims

1. A skeleton action recognition method based on spatiotemporal topology learning, characterized in that: The skeleton data to be processed is acquired, input into a pre-trained skeleton-based action classification recognition network model, and an action classification result of the skeleton data to be processed is obtained; including the following steps: S1, pre-processing the input three-dimensional skeleton joint sequence data, including denoising, standardization and normalization processing; S2, extracting the spatio-temporal topological features of the skeleton sequence, constructing a two-dimensional topological coding module, constructing a static spatial relationship code by using a shortest path graph weight, constructing a dynamic spatial relationship code based on a persistent barcode graph sequence, and analyzing the static relationship of the skeleton structure and the dynamic topological change in the action process; S3, constructing a multi-level spatial attention aggregation module based on hierarchical skeleton features, learning the feature weights of key action parts at multiple structural scales through a representative feature extractor and edge convolution operation, and realizing multi-scale structural saliency enhancement; S4, designing a time dynamic sequence pooling method, embedding the time sequence features into a high-order model through the theory of Kupman operator and dynamic mode decomposition, capturing the time dynamics law in the nonlinear action sequence by constructing a dynamic matrix, and combining a SoftMax classifier for class recognition.

2. The spatio-temporal topology learning based skeletal action recognition method of claim 1, wherein: Step S2 extracts the spatio-temporal topological features of the skeleton sequence and constructs a two-dimensional topological coding module, specifically: S2.1, constructing a static spatial coding component: by using graph distance measurement to quantify the relative position between joint pairs in the skeleton graph, an anatomical skeleton connection model is established, and the static coding formula is as follows: where M e Path(G s ) is the set of all paths in the skeleton graph G s from vertex v i to vertex v j , |E| is the number of edges in the path E, d ij is the shortest path distance from v i to v j , and M1 and M |E| denote the first and last vertices of the path M, respectively. to convert the shortest path distance d i,j into a weight D ij ; a weight parameter D ij is defined for each pair of joints in the skeleton graph by mapping the shortest path distance d ij between the joint pair into the weight parameter; a weight is assigned to each pair of joints, which is subsequently used to update the adjacency matrix in the skeleton graph to preserve the skeletal connectivity information; S2.2, Constructing dynamic spatial coding component: a weighted dynamic graph D is constructed by given input pose sequence G , where the skeleton joints are nodes and the Euclidean distance between joint pairs are edge weights ω ij , according to whether ω ij exceeds threshold value ∈, the edges in the graph are retained or deleted, and ∈ is initially set to 1; the dynamic graph D G is filtered using persistent topology analysis method, and the filtering is considered instead of D G as a single object, and the formula is as follows: The formula represents the process of starting from an empty set and gradually adding simplices until a complete simplex complex S is constructed; wherein i = 1, 2, … m represents the abstract simplex complex of each graph; each S i is regarded as a subgraph or subcomplex of D G , and a new simplex is added at each step according to the previous simplex and the newly introduced vertex or edge; the simplicial complex is constructed by adopting the Vietoris-Rips complex, which determines whether an edge exists between vertices according to the distance between them, thereby constructing a simplicial complex from the graph; the generation and disappearance of topological features are captured; these dynamic topological structures are encoded into persistent barcodes, which summarize the temporal appearance and disappearance of structural patterns; by applying the differentiable vectorization function Ψ 0 , the barcodes are converted into fixed-length vectors; then, through a linear projection f θ , a set of dynamic topological features K is generated and embedded into the feature space of the graph convolution network, and the specific formula is as follows: where f θ is a linear mapping layer defined by the parameter θ, which projects the output of the vectorization function Ψ 0 into the feature space of the graph convolution network hidden layer; Ψ 0 is responsible for converting the persistent barcode into a fixed-length feature vector; represents a 0-dimensional homology persistent barcode, which encodes the persistence of connected components in the graph; finally, the dynamic topological feature K is embedded into the feature space of the graph convolution network, providing additional information about the specific action topological changes to the model.

3. The spatio-temporal topology learning based skeletal action recognition method of claim 1, wherein: Step S1 pre-processes the input three-dimensional skeleton joint sequence data, specifically: The input skeleton sequence data is normalized to scale each joint coordinate value in the skeleton sequence data to a pre-set range; at the same time, the skeleton sequence data is denoised to remove noise data caused by sensor errors or environmental interference; In addition, the skeleton sequence data is time-aligned to ensure that different action sequences have consistent starting points and ending points in the time dimension.

4. The spatio-temporal topology learning based skeletal action recognition method of claim 1, wherein: Step S3 uses a multi-level spatial attention aggregation module to weight and aggregate the skeleton node features at different levels, specifically: A representative feature extractor is used to identify key representative nodes in each level; the node features of each level are collected to obtain a representative feature set of each level, as shown in the formula below: wherein, Ψ represents a representative feature extraction operation, first extracting the features of the representative nodes in each level, and then performing spatial average pooling on these features; F (m) represents the features of the mth level, T m represents the total number of nodes in the node set K m at the mth level, T m+1 represents the total number of nodes in the node set K m+1 at the m+1th level, K m represents the node set at the mth level, K m+1 represents the node set at the m+1th level; represents taking the maximum value of the features of node v in the time dimension t, that is, selecting the maximum feature value of the node at all time steps; then summing all nodes in K m and K m+1 , and dividing by T m +T m+1 ; All levels of features are regarded as nodes in the graph, and the similarity of nodes in the hierarchical feature space is learned according to the Euclidean distance between nodes; the final attention is the result of the edge convolution operation activated by the sigmoid function, as shown in the formula below: The function Ψ represents the operation applied to each hierarchical feature F (m) The operator N represents the edge convolution operation applied at each hierarchy, and The symbol || represents the edge convolution operation on the merged features, and the sigmoid function σ is used to map the features to the range between 0 and 1, thereby representing the importance of each hierarchical feature; finally, the attention map is weighted and added to the feature map obtained by the convolution output to generate the final output feature map.

5. The spatio-temporal topology learning based skeletal action recognition method according to claim 1, characterized in that: Step S4 designs a time dynamic sequence pooling, constructs a time sequence feature dynamic matrix through Kupman theory, and extracts high-order dynamic information in the time dimension, specifically: Nonlinear systems are analyzed by linear transformations in high-dimensional spaces via Koopman operators; Koopman operators are approximated in an efficient computational form using dynamic mode decomposition; each feature vector x t is first embedded into a higher-dimensional space by an embedding function ψ, and the transition between states is approximated as a linear equation whose formula is as follows: ψx t+1 = (Kψ)(x t ) where K denotes the Koopman operator; a linear approximation matrix D is constructed using dynamic mode decomposition, which controls the temporal dynamics of the sequence; a feature matrix X is given for a continuous time step 1:m-1 and X 2:m The transition matrix formula inspired by the Koopman operator is: where X 1:m-1 is the feature matrix for time step 1 to m-1, X 2:m is the feature matrix for time step 2 to m, denotes the pseudo-inverse; the matrix D captures the essential dynamic behavior of the sequence and serves as a pooled representation of the temporal structure; for classification, each action class is assigned a learnable matrix D that encodes its unique temporal signature, and for a given sequence, D i · X t The evolution state is computed and compared to the next state X q by the Euclidean distance, as follows: s i = -d i s i is the negation of the distance and is considered as a classification score, and the final prediction is made by SoftMax on all class scores: where p i is the probability of the ith class, and N is the total number of classes.

Images