Spatial-temporal topology learning-based skeleton action recognition method

Abstract

The invention relates to a skeleton action recognition method based on spatio-temporal topology learning, and belongs to the technical field of computer vision and artificial intelligence. The method comprises the following steps: constructing a two-dimensional topological coding module to capture a static topological relation and a dynamic topological relation of a skeleton structure at the same time; a multi-level space attention aggregation module is adopted to carry out adaptive weighted aggregation on skeleton features of different levels so as to enhance the expression ability of key nodes and edges; a time dynamic sequence pooling method is designed, high-order dynamic information of time sequence data is extracted based on a dynamic mode decomposition strategy, and the accuracy of action recognition is improved. The method has the advantages that time-space structure information of skeleton data can be fully utilized, accurate recognition of human body actions is achieved, meanwhile, the modeling capacity of complex dynamic behaviors is improved, and the limitation of a traditional method in capturing high-order dynamic information is avoided.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence technologies, and in particular to a skeletal motion recognition method based on spatiotemporal topological learning, which aims to improve the ability to model the spatiotemporal features of human skeletal data and provide an end-to-end deep learning framework to achieve accurate recognition of complex motions. Background Art

[0002] The human skeleton is a dynamic structure composed of multiple joints and their connections. Traditional modeling methods typically rely on simple sequences of joint coordinates, ignoring the complex topological features of the skeleton. Despite significant progress in skeletal action recognition, existing methods still face numerous challenges. First, existing skeletal action recognition methods fail to fully utilize the topological information in skeletal data. For example, actions such as jumping and waving involve not only the movement of a single joint but also the coordinated relationships between multiple joints. Existing methods struggle to accurately model these dependencies. Second, existing methods typically employ a fixed skeletal topology, making it difficult to adapt to individual variations in the skeleton and resulting in insufficient generalization. Furthermore, while some methods incorporate learnable topologies, this can lose the physical constraints of the original skeleton, making the model's learning of topological relationships unstable. Finally, there are still deficiencies in modeling temporal information. Current mainstream approaches include strategies based on recurrent neural networks (RNNs), graph convolutional networks (GCNs), and temporal convolutional networks (TCNs). While these methods can capture temporal features to a certain extent, most only process low-order temporal information and are unable to effectively model the high-order dynamic features of complex actions. For example, in movements such as running and rotation, the temporal changes in the movements are not just a simple accumulation of joint movements, but rather have complex temporal dependencies. Traditional methods mainly rely on temporal pooling (such as maximum pooling or average pooling) for global temporal feature aggregation, which makes it difficult to extract true high-order temporal dynamic information, thus limiting the ability to recognize complex movements. Therefore, there is an urgent need for a model framework that can simultaneously solve key problems such as spatial topological modeling of skeletal data, temporal dynamic feature extraction, and adaptive topological learning, while meeting efficient computing requirements, accurately modeling complex motion relationships, and improving generalization capabilities in different scenarios.

[0003] In summary, in order to achieve accurate recognition of complex human movements, innovations are urgently needed in the spatial topological modeling of skeletal data, temporal dynamic feature extraction, and adaptive topological learning. Summary of the Invention

[0004] The purpose of the present invention is to provide a skeletal motion recognition method based on spatiotemporal topological learning, which solves the technical problems in the prior art that the topological structure of skeletal data is not fully utilized, the dynamic relationship between joints is difficult to accurately model, and the high-order temporal information modeling capability is lacking. The present invention systematically solves the limitations of existing methods in topological modeling, temporal modeling and feature extraction by constructing a two-dimensional topological encoding module, a multi-level spatial attention aggregation module and a temporal dynamic sequence pooling method, and significantly improves the accuracy and generalization ability of motion recognition. The present invention successfully improves the recognition accuracy of human motion and is extremely suitable for intelligent monitoring, autonomous driving, virtual reality, human-computer interaction, medical rehabilitation and other fields, and provides an efficient, accurate and robust solution for complex motion recognition tasks.

[0005] The main idea of the construction method of the skeletal action recognition based on spatiotemporal topology learning of the present invention is: for the input human skeleton data sequence, first, the skeletal structure is jointly modeled with static topology and dynamic topology through a two-dimensional topological coding module, the inherent structural characteristics of the human skeleton are maintained by static spatial relationship coding, and the change information of the temporal topology during the joint motion is extracted through dynamic spatial relationship coding to enhance the structural expression ability of the skeletal data. At the same time, a multi-level spatial attention aggregation module is introduced to perform adaptive weighted aggregation for skeletal features of different levels, so that the model can effectively focus on key joints and important motion patterns, and improve the accuracy of feature expression. In addition, the present invention adopts a time dynamic sequence pooling method, based on dynamic mode decomposition theory, to extract high-order dynamic information in the time series in a more efficient way, making up for the shortcomings of the traditional time pooling method in time-dependent modeling. Finally, accurate action recognition is achieved on skeletal data, while possessing a stronger spatiotemporal feature capture capability.

[0006] The above-mentioned purpose of the present invention is achieved through the following technical solutions:

[0007] The skeleton action recognition method based on spatiotemporal topology learning obtains skeleton data to be processed, inputs it into a pre-trained skeleton-based action classification and recognition network model, and obtains the action classification result of the skeleton data to be processed; the method includes the following steps:

[0008] S1. Skeletal data preprocessing: Preprocess the input 3D skeletal joint sequence data using methods including denoising, standardization, and normalization to ensure the stability and consistency of the skeletal data.

[0009] S2. The first stage of the skeletal action recognition model: extracting the spatiotemporal topological features of the skeletal sequence, constructing a two-dimensional topological encoding module, constructing a static spatial relationship encoding by adopting the shortest path graph weight, and constructing a dynamic spatial relationship encoding using a graph sequence based on persistent barcodes, analyzing the static relationship of the skeletal structure and the dynamic topological changes during the action process;

[0010] S3, the second stage of the skeletal action recognition model: a multi-level spatial attention aggregation module is constructed based on hierarchical skeleton features. The feature weights of key action parts are learned at multiple structural scales through representative feature extractors and edge convolution operations, achieving multi-scale structural saliency enhancement.

[0011] S4. The third stage of the skeletal action recognition model: Design a temporal dynamic sequence pooling method, perform high-order modeling of the temporal feature embedding vector through Koopman operator theory and dynamic pattern decomposition, construct a dynamic matrix to capture the temporal dynamics in nonlinear action sequences, and combine it with the SoftMax classifier for category recognition.

[0012] The skeleton data preprocessing described in step S1 is specifically:

[0013] The input skeletal sequence data is normalized, and the coordinate value of each joint in the skeletal sequence data is scaled to a preset range to eliminate the scale differences between different data sources; at the same time, the skeletal sequence data is denoised to remove noise data caused by sensor errors or environmental interference to improve the quality and stability of the data; in addition, the skeletal sequence data is time-aligned to ensure that different action sequences have consistent starting and ending points in the time dimension to facilitate subsequent feature extraction and analysis.

[0014] Step S2 extracts the spatiotemporal topological features of the skeleton sequence and constructs a two-dimensional topological encoding module. The specific method is as follows:

[0015] S2.1. Constructing a Static Spatial Encoding Component: Designing a static spatial encoding component to analyze the static relationships of the skeletal structure. By utilizing a graph distance metric to quantify the relative positions of joint pairs in the skeletal graph, we can clearly establish an anatomical skeletal connection model. First, we establish the relative positions of joint pairs in the skeletal graph using the following formula:

[0016]

[0017] Among them, M∈Path(G s ) is the skeleton graph G s From vertex v i To the vertex v j is the set of all paths of , |E| is the number of edges in path E, d ij It is from v i to v j The shortest path distance, M1 and M |E| Represent the first and last vertices of the path M respectively, Used to convert the shortest path distance d i,j Convert to weight D ij, define a weight parameter D for each pair of joints in the skeleton graph ij , by taking the shortest path distance d between the joint pairs ij , mapped into weight parameters, assigning a weight to each pair of joints. The weights are then used to update the adjacency matrix in the skeleton graph to preserve the connection information between bones. Finally, the statically encoded topological encoding is added to the hidden features of GCN to capture the basic structural information of the skeleton;

[0018] S2.2. Constructing dynamic spatial encoding components: Designing dynamic spatial encoding components to analyze the dynamic topological relationship of the skeletal structure; given an input posture sequence, constructing a weighted dynamic graph D G , with skeleton joints as nodes and the Euclidean distance between joint pairs as edge weights, denoted as ω ij According to ω ij Whether the threshold ∈ is exceeded, the edge in the graph is retained or deleted, and the initial value of ∈ is set to 1; the persistent topology analysis method is used to analyze the dynamic graph D G For filtering, the main idea is to consider filtering Instead of D G Considered as a single object, the formula is as follows:

[0019]

[0020] This formula represents the process of starting from an empty set and gradually adding simplexes until a complete simplex complex number S is constructed; i=1,2,…m represents the abstract simplex complex of each graph, where each S i Can be regarded as D G The subgraph or subcomplex of a graph is constructed by taking the Vietoris-Rips complex and adding new simplices at each step based on the previous simplices and the newly introduced vertices or edges; the simplicial complex is constructed from the graph by taking the Vietoris-Rips complex and determining whether there is an edge based on the distance between vertices; the generation and disappearance of topological features (such as connected components) are captured; these dynamic topological structures are encoded into persistent barcodes that summarize the temporal emergence and disappearance of structural patterns; the differentiable vectorization function Ψ is applied to the graph to obtain the topological structure of the graph. 0 These barcodes are converted into fixed-length vectors; then a linear projection f θ Generate a set of dynamic topological features K and embed them into the feature space of the graph convolutional network. The specific formula is as follows:

[0021]

[0022] Among them, f θ is a linear mapping layer defined by parameter θ, which vectorizes the function Ψ 0 The output of is projected into the feature space of the hidden layer of the graph convolutional network;0 Responsible for the persistent barcode Convert to a fixed-length feature vector; It represents a 0-dimensional homology persistence barcode that encodes the persistence of connected components in the graph. Finally, the dynamic topological feature K is embedded into the feature space of the graph convolutional network to provide the model with additional information about the topological changes of specific actions.

[0023] In step S3, the weighted aggregation method using the multi-level spatial attention aggregation module is as follows:

[0024] S3.1. Representative Feature Extractor: The skeleton data calculated in step S1 and the static and dynamic mop structures calculated in step S2 are used to identify the key representative nodes in each layer by using a representative feature extractor. The node features of each layer are concentrated to obtain the representative feature set of each layer. The formula is as follows:

[0025]

[0026] Where Ψ represents the representative feature extraction operation, first extracting the features of the representative nodes in each layer, and then performing spatial average pooling on these features. (m) represents the features of the mth level, T m Represents the node set K at the mth level m The total number of nodes in T m+1 Represents the node set K at the m+1th level m+1 The total number of nodes in K m represents the node set at the mth level, K m+1 Represents the node set at the m+1th level. Indicates taking the maximum value of the feature of node v in the time dimension t, that is, selecting the maximum feature value of the node in all time steps. Then K m and K m+1 Sum all nodes in and divide by T m +T m+1 ;

[0027] S3.2, Edge Convolution: After the representative feature extraction layer, since the features at each level do not yet share information with each other, it is necessary to further integrate these feature maps. The features of all levels are regarded as nodes in the graph, and the similarity of nodes in the hierarchical feature space is learned based on the Euclidean distance between nodes. The final attention is the result of the edge convolution operation activated by the sigmoid function, which is expressed as follows:

[0028]

[0029] Function Ψ represents the function applied to each level feature F (m)The operator N represents the edge convolution operation applied at each level, and It represents the merging of features at all levels, the symbol ‖ represents the marginal convolution operation on the merged features, and the sigmoid function σ is used to map the features to the range between 0 and 1, thereby indicating the importance of features at each level.

[0030] Finally, the attention map is weighted and added to the feature map obtained by the convolution output to generate the final output feature map.

[0031] The method for designing the time dynamic sequence pooling method in step S4 to extract high-order dynamic information in the time dimension is as follows:

[0032] The nonlinear system is analyzed by linear transformation in high-dimensional space through the Koopman operator; the Koopman operator is approximated in a computationally efficient form using dynamic mode decomposition; each eigenvector x is first embedded in the function ψ t Embedded in a higher-dimensional space, the transition between states is approximated by a linear equation, which is formulated as follows:

[0033] ψx t+1 =(Kψ)(x t )

[0034] Where K represents the Koopman operator. Dynamic mode decomposition is used to construct a linear approximation matrix D, which controls the temporal dynamics of the sequence; given the feature matrix X of consecutive time steps 1:m-1 and X 2:m , the transfer matrix formula inspired by the Koopman operator is:

[0035]

[0036] Among them, X 1:m-1 is the feature matrix of time steps 1 to m-1, X 2:m is the feature matrix from time steps 2 to m, Represents the pseudo-inverse matrix; the matrix D captures the basic dynamic behavior of the sequence and is used as a pooled representation of the temporal structure; for classification, each action category is assigned a learnable matrix D that encodes its unique temporal features. For a given sequence, D is used i ·X t Calculate the evolved state and compare it to the next state X by Euclidean distance q For comparison, the formula is as follows:

[0037]

[0038] s i =-d i

[0039] s i It is the inverse of the distance and is regarded as the classification score. The final prediction of all class scores is made through SoftMax:

[0040]

[0041] where p i is the probability of the ith class, and N is the total number of classes.

[0042] The beneficial effects of the present invention are:

[0043] 1. The skeletal action recognition method based on spatiotemporal topological learning proposed in this paper breaks through the limitations of existing methods and innovatively constructs a two-dimensional topological coding module. By combining static spatial relationship coding with dynamic spatial relationship coding, it fully mines the static topological structure and dynamic motion pattern of skeletal data. Compared with traditional graph convolutional network methods, this method can more accurately model skeletal topological relationships, avoid the loss of topological information, and thus improve the accuracy and stability of action recognition.

[0044] 2. A multi-level spatial attention aggregation module is used to effectively enhance the ability to extract key skeletal features. Through a multi-scale attention mechanism, the present invention can aggregate skeletal node features at different levels, highlight key action parts, and improve the ability to model dynamic information of the human skeleton. Compared with traditional pooling or average aggregation methods, the multi-level spatial attention aggregation module can adaptively adjust the feature weights of different parts, making the model more robust in complex action recognition tasks.

[0045] 3. A time dynamic sequence pooling method is proposed to capture the high-order dynamic information of time series data and improve the time modeling capability. This invention uses the Koopman theory to construct a time series feature dynamic matrix, breaking through the limitation of the traditional time pooling method that can only capture first-order time information, improving the recognition ability of long-span actions, and making the system more advantageous when processing continuous actions (such as dance, sports, etc.). Compared with the traditional graph convolutional network structure, this invention reduces redundant calculations and improves computational efficiency, so that it can still maintain efficient reasoning capabilities on large-scale data sets.

[0046] 4. Applicable to a variety of application scenarios, with good generalization and practical value. The method of the present invention not only achieves excellent recognition performance on public action recognition datasets (NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA), but is also applicable to multiple fields such as intelligent security, behavior analysis, human-computer interaction, and medical rehabilitation. Its good generalization ability allows the method to be extended to different types of skeletal data, improving its applicability and flexibility in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide further understanding of the present invention and constitute a part of this application. The illustrative examples of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0048] Figure 1 A process framework diagram for establishing the present invention;

[0049] Figure 2 A diagram of the steps for establishing the present invention;

[0050] Figure 3 This is a diagram of the skeleton action recognition network structure based on spatiotemporal topology learning of the present invention;

[0051] Figure 4 This is a structural diagram of the dual-dimensional topological encoding component of the present invention;

[0052] Figure 5 This is a structural diagram of the multi-scale attention aggregation component of the present invention;

[0053] Figure 6 This is a diagram of the time dynamic sequence pooling structure of the present invention;

[0054] Figure 7 This is a visualization result diagram of t-SNE on the NTU RGB+D 60 dataset in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0056] See also Figures 1 to 7 As shown, the skeleton action recognition method based on spatiotemporal topological learning of the present invention first preprocesses the original data, including data normalization and coordinate adjustment. The training set and the validation set are divided into 8:2 divisions. The training set is used for training, and the error between the actual output result of the training and the label value is calculated. The difference is transferred from top to bottom through the back propagation algorithm, and the weight is updated using the weight. After training, the trained neural network model is saved, and the validation set is input to adjust the parameters to make a preliminary evaluation of the training status of the network model. Specifically, the following steps are included:

[0057] S1. Skeletal Data Preprocessing: The input skeletal sequence data is normalized, scaling each joint coordinate value within a preset range to eliminate scale differences between different data sources. Simultaneously, the skeletal sequence data is subjected to noise reduction, removing noise caused by sensor errors or environmental interference to improve data quality and stability. Furthermore, the skeletal sequence data is time-aligned to ensure that different action sequences have consistent start and end points in the temporal dimension, facilitating subsequent feature extraction and analysis.

[0058] S2. The first stage of the skeletal action recognition model: extracting the spatiotemporal topological features of the skeletal sequence, constructing a two-dimensional topological encoding module, constructing a static spatial relationship encoding by adopting the shortest path graph weight, and constructing a dynamic spatial relationship encoding using a graph sequence based on persistent barcodes, analyzing the static relationship of the skeletal structure and the dynamic topological changes during the action process;

[0059] S2.1. Design of static spatial coding components:

[0060] The static spatial encoding component explicitly models the anatomical skeletal connections by utilizing a graph distance metric to quantify the relative positions between joint pairs in the skeletal graph. The static encoding formula is as follows:

[0061]

[0062] Among them, M∈Path(G s ) is the skeleton graph G s From vertex v i To the vertex v j is the set of all paths of , |E| is the number of edges in path E, d ij It is from v i to v j The shortest path distance, M1 and M |E| Represent the first and last vertices of the path M respectively, Used to convert the shortest path distance d i,j Convert to weight D ij ; Define a weight parameter D for each pair of joints in the skeleton graph ij , by taking the shortest path distance d between the joint pairs ij , mapped into weight parameters, assigning a weight to each pair of joints. This weight is then used to update the adjacency matrix in the skeleton graph to preserve the bone connection information.

[0063] S2.2. Design of dynamic spatial coding components:

[0064] Construct a weighted dynamic graph D given an input pose sequence G, where the skeleton joints are nodes and the Euclidean distance between joint pairs is the edge weight ω ij , according to ω ij Whether the threshold ∈ is exceeded, the edge in the graph is retained or deleted, and the initial value of ∈ is set to 1. The persistent topology analysis method is used to analyze the dynamic graph D G For filtering, the main idea is to consider filtering Instead of D G Considered as a single object, the formula is as follows:

[0065]

[0066] The formula represents the process of starting from an empty set and gradually adding simplexes until a complete simplex complex number S is constructed. i=1,2,…m represents the abstract simplex complex of each graph, where each S i Can be regarded as D G The subgraph or subcomplex of the graph is constructed by taking the Vietoris-Rips complex and determining whether an edge exists based on the distance between vertices. These dynamic topological structures are encoded into a persistent barcode that summarizes the temporal appearance and disappearance of structural patterns. 0 These barcodes are converted into fixed-length vectors; then a linear projection f θ Generate a set of dynamic topological features K and embed them into the feature space of the graph convolutional network. The specific formula is as follows:

[0067]

[0068] Among them, f θ is a linear mapping layer defined by parameter θ, which vectorizes the function Ψ 0 The output of is projected into the feature space of the hidden layer of the graph convolutional network; 0 Responsible for the persistent barcode Convert to a fixed-length feature vector; It represents a 0-dimensional homology persistence barcode that encodes the persistence of connected components in the graph. Finally, the dynamic topological feature K is embedded into the feature space of the graph convolutional network to provide the model with additional information about the topological changes of specific actions.

[0069] S3, using a multi-level spatial attention aggregation module to perform weighted aggregation on the features of bone nodes at different levels, enhance key bone edge information, and optimize feature expression capabilities;

[0070] S3.1. First, use the representative feature extractor to identify the key representative nodes in each layer:

[0071] The skeleton data calculated by the S1 process and the static and dynamic mop structures calculated by the S2 process are used to identify the key representative nodes in each layer by using a representative feature extractor. The node features of each layer are concentrated to obtain the representative feature set of each layer. The formula is as follows:

[0072]

[0073] Ψwhere Ψ represents the representative feature extraction operation, first extracting the features of the representative nodes in each layer, and then performing spatial average pooling on these features. (m) represents the features of the mth level, T m Represents the node set K at the mth level m The total number of nodes in T m+1 Represents the node set K at the m+1th level m+1 The total number of nodes in K m represents the node set at the mth level, K m+1 Represents the node set at the m+1th level. Indicates taking the maximum value of the feature of node v in the time dimension t, that is, selecting the maximum feature value of the node in all time steps. Then K m and K m+1 Sum all nodes in and divide by T m +T m+1 ;

[0074] S3.2. Using edge convolution to learn node similarity in hierarchical feature space:

[0075] After the representative feature extraction layer, since the features at each level do not yet share information with each other, it is necessary to further integrate these feature maps. The features of all levels are regarded as nodes in the graph, and the similarity of nodes in the hierarchical feature space is learned based on the Euclidean distance between nodes; the final attention is the result of the edge convolution operation activated by the sigmoid function, which is expressed as follows:

[0076]

[0077] Function Ψ represents the function applied to each level feature F (m) The operator N represents the edge convolution operation applied at each level, and It represents the merging of features at all levels, the symbol ‖ represents the marginal convolution operation on the merged features, and the sigmoid function σ is used to map the features to the range between 0 and 1, thereby indicating the importance of features at each level.

[0078] Finally, the attention map is weighted and added to the feature map obtained by the convolution output to generate the final output feature map.

[0079] S4. Design a time dynamic sequence pooling method to extract high-order dynamic information in the time dimension, and finally complete the recognition of the entire skeleton behavior action.

[0080] The nonlinear system is analyzed by linear transformation in high-dimensional space through the Koopman operator; the Koopman operator is approximated in a computationally efficient form using dynamic mode decomposition; each eigenvector x is first embedded in the function ψ t Embedded in a higher-dimensional space, the transition between states is approximated by a linear equation, which is formulated as follows:

[0081] ψx t+1 =(Kψ)(x t )

[0082] Where K represents the Koopman operator. Dynamic mode decomposition is used to construct a linear approximation matrix D, which controls the temporal dynamics of the sequence; given the feature matrix X of consecutive time steps 1:m-1 and X 2:m , the transfer matrix formula inspired by the Koopman operator is:

[0083]

[0084] Among them, X 1:m-1 is the feature matrix of time steps 1 to m-1, X 2:m is the feature matrix from time steps 2 to m, Represents the pseudo-inverse matrix; the matrix D captures the basic dynamic behavior of the sequence and is used as a pooled representation of the temporal structure; for classification, each action category is assigned a learnable matrix D that encodes its unique temporal features. For a given sequence, D is used i ·X t Calculate the evolved state and compare it to the next state X by Euclidean distance q For comparison, the formula is as follows:

[0085]

[0086] s i =-d i

[0087] s i It is the inverse of the distance and is regarded as the classification score. The final prediction of all class scores is made through SoftMax:

[0088]

[0089] where p i is the probability of the ith class, and N is the total number of classes.

[0090] S5. Define the network model loss function, and train and verify the network structure of the present invention in the training set and verification set through multiple iterations. to penalize classification errors.

[0091] The proposed network structure is named TE-STGCN. To fully evaluate the performance of TE-STGCN in skeletal action recognition, three mainstream large-scale skeletal action recognition datasets, NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA, were selected as evaluation benchmarks. Experiments were conducted under cross-subject (X-Sub), cross-view (X-View), and cross-setting (X-Set) protocols. Comparative analysis was performed with current mainstream advanced methods to verify the effectiveness and superiority of the proposed method.

[0092] 1. Dataset Introduction

[0093] The NTU RGB+D 60 dataset is a large-scale skeletal action recognition dataset consisting of 60 action categories performed by 40 subjects, totaling 56,880 skeletal sequences. The data was collected using a multi-view RGB-D sensor and covers three categories: daily activities, health-related actions, and social interactions. The dataset contains three-dimensional skeletal joint coordinate information, with each skeleton consisting of 25 joint points. During the experiments, the official dataset partitioning method was strictly followed, using two evaluation protocols: cross-subject (X-Sub) and cross-view (X-View). In the X-Sub scheme, the 40 subjects were randomly divided into training and test sets, while in the X-View scheme, the subjects were divided according to camera angle.

[0094] The NTU RGB+D 120 dataset is an extension of the NTU RGB+D 60 dataset, adding 60 new action categories, for a total of 120 categories, and containing 114,480 skeletal action sequences. Experiments on this dataset were conducted under both cross-subject (X-Sub) and cross-setting (X-Set) evaluation protocols. The X-Set scheme uses a split between even-numbered and odd-numbered ID categories for training and testing.

[0095] 2. Evaluation Metrics

[0096] In order to quantitatively analyze the performance of the proposed TE-STGCN network, the Top-1 accuracy is used as the evaluation indicator for action recognition, and the calculation formula is as follows:

[0097]

[0098] Among them, TP (True Positive) represents correctly classified action samples, FP (False Positive) represents misclassified action samples, and FN (False Negative) represents incorrectly identified action samples.

[0099] In addition, the present invention uses t-SNE visualization to visualize the classification ability of the model, observe the model's ability to distinguish similar action categories, and calculate the misclassification rate between categories.

[0100] 3. Experimental results

[0101] NTU RGB+D 60 dataset: As shown in Table 1, the TE-STGCN network of the present invention achieves a Top-1 recognition accuracy of 92.3% under the X-Sub scheme and a Top-1 recognition accuracy of 96.7% under the X-View scheme, which is an improvement of 1.1% and 1.4% compared to CTR-GCN. In addition, the method of the present invention achieves better recognition performance in 48 of the 60 action categories, especially in actions with large motion amplitudes (such as "jumping" and "waving") and subtle actions (such as "writing" and "drinking water").

[0102] Table 1: Quantitative results on the NTU RGB+D 60 dataset

[0103]

[0104]

[0105] NTU RGB+D 120 dataset: As shown in Table 2, TE-STGCN achieves a Top-1 recognition accuracy of 88.3% under the X-Sub scheme and 89.5% under the X-Set scheme, which are 1.4% and 1.2% higher than CTR-GCN, respectively.

[0106] Table 2: Quantitative results on the NTU RGB+D 120 dataset

[0107]

[0108]

[0109] See also Figure 7The t-SNE visualization results of the feature space distribution of representative action classes under the X-View protocol of the NTU RGB+D 60 dataset are shown in the figure. Four different action categories were randomly selected for comparative analysis: "combing hair" (blue), "falling" (orange), "sitting" (green) and "putting on shoes" (red). The present invention generates compact, well-separated feature clusters, demonstrating enhanced action discrimination capabilities. The intra-class feature consistency is improved, ensuring that instances of the same action remain close in the feature space. This improved geometric arrangement in the feature space indicates that the model's ability to distinguish fine-grained motion patterns has been enhanced. These visualization results confirm that the model of the present invention generates compact, well-separated feature clusters, ensuring high consistency within the class and distinguishability between classes.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements to the present invention are intended to fall within the scope of protection of the present invention.

Claims

1. A skeletal action recognition method based on spatiotemporal topological learning, characterized by: Obtaining the skeleton data to be processed and inputting it into a pre-trained skeleton-based motion classification and recognition network model to obtain the motion classification result of the skeleton data to be processed; comprising the following steps: S1. Preprocessing the input 3D skeletal joint sequence data, including denoising, standardization and normalization; S2. Extract the spatiotemporal topological features of the skeletal sequence and construct a two-dimensional topological encoding module. This module uses the shortest path graph weight to construct a static spatial relationship encoding and uses a persistent barcode-based graph sequence to construct a dynamic spatial relationship encoding. This module analyzes the static relationship of the skeletal structure and the dynamic topological changes during movement. S3: Building a multi-level spatial attention aggregation module based on hierarchical skeleton features. Through representative feature extractors and edge convolution operations, the feature weights of key action parts are learned at multiple structural scales to achieve multi-scale structural saliency enhancement. S4. Design a temporal dynamic sequence pooling method, perform high-order modeling of temporal feature embedding vectors through Koopman operator theory and dynamic pattern decomposition, construct a dynamic matrix to capture the temporal dynamics in nonlinear action sequences, and combine it with the SoftMax classifier for category recognition.

2. The skeletal motion recognition method based on spatiotemporal topology learning according to claim 1, wherein: The step S2 extracts the spatiotemporal topological features of the skeleton sequence and constructs a two-dimensional topological encoding module, specifically: S2.

1. Constructing a static spatial encoding component: We explicitly model the anatomical skeletal connections by utilizing a graph distance metric to quantify the relative positions of joint pairs in the skeletal graph. The static encoding formula is as follows: Among them, M∈Path(G s ) is the skeleton graph G s From vertex v i To the vertex v j is the set of all paths of , |E| is the number of edges in path E, d ij It is from v i to v j The shortest path distance, M1 and M |E| Represent the first and last vertices of the path M respectively, Used to convert the shortest path distance d i,j Convert to weight D ij ; Define a weight parameter D for each pair of joints in the skeleton graph ij , by taking the shortest path distance d between the joint pairs ij , mapped into weight parameters; assigning a weight to each pair of joints, which is then used to update the adjacency matrix in the skeleton graph to preserve the skeleton connection information; S2.

2. Constructing a dynamic spatial encoding component: A weighted dynamic graph D is constructed by giving a given input pose sequence. G , where the skeleton joints are nodes and the Euclidean distance between joint pairs is the edge weight ω ij , according to ω ij Whether the threshold ∈ is exceeded, the edge in the graph is retained or deleted, and the initial value of ∈ is set to 1; the persistent topology analysis method is used to analyze the dynamic graph D G Filtering, consider filtering Instead of D G Considered as a single object, the formula is as follows: The formula represents the process of starting from an empty set and gradually adding simplexes until a complete simplex complex number S is constructed; i=1,2,…m represents the abstract simplex complex of each graph; each S i All are considered D G The method constructs a subgraph or subcomplex of a graph, where each step adds a new simplicial based on the previous simplicial and the newly introduced vertex or edge; constructs a simplicial complex from a graph by taking the Vietoris-Rips complex and determining whether an edge exists based on the distance between vertices; captures the generation and disappearance of topological features; these dynamic topological structures are encoded into a persistent barcode that summarizes the temporal emergence and disappearance of structural patterns; and applies a differentiable vectorized function Ψ 0 The barcode is converted into a fixed-length vector; then a linear projection f θ Generate a set of dynamic topological features K and embed them into the feature space of the graph convolutional network. The specific formula is as follows: Among them, f θ is a linear mapping layer defined by parameter θ, which vectorizes the function Ψ 0 The output of is projected into the feature space of the hidden layer of the graph convolutional network; 0 Responsible for the persistent barcode Convert to a fixed-length feature vector; It represents a 0-dimensional homology persistence barcode that encodes the persistence of connected components in the graph. Finally, the dynamic topological feature K is embedded into the feature space of the graph convolutional network to provide the model with additional information about the topological changes of specific actions.

3. The skeletal motion recognition method based on spatiotemporal topology learning according to claim 1, wherein: Step S1 pre-processes the input three-dimensional skeleton joint point sequence data, specifically: Normalize the input skeleton sequence data and scale the coordinate value of each joint in the skeleton sequence data to a preset range; at the same time, perform noise reduction on the skeleton sequence data to remove noise data caused by sensor errors or environmental interference; In addition, the skeleton sequence data is time-aligned to ensure that different action sequences have consistent starting and ending points in the time dimension.

4. The skeletal motion recognition method based on spatiotemporal topology learning according to claim 1, wherein: Step S3 uses a multi-level spatial attention aggregation module to perform weighted aggregation on the skeleton node features at different levels. Specifically: A representative feature extractor is used to identify the key representative nodes in each layer; the node features of each layer are concentrated to obtain the representative feature set of each layer. The formula is as follows: Among them, Ψ represents the representative feature extraction operation, first extracting the features of the representative nodes in each layer, and performing spatial average pooling on these features; F (m) represents the features of the mth level, T m Represents the node set K at the mth level m The total number of nodes in T m+1 Represents the node set K at the m+1th level m+1 The total number of nodes in K m represents the node set at the mth level, K m+1 Represents the node set at the m+1th level; Indicates taking the maximum value of the feature of node v in the time dimension t, that is, selecting the maximum feature value of the node in all time steps; then K m and K m+1 Sum all nodes in and divide by T m +T m+1 ; The features of all levels are regarded as nodes in the graph, and the similarity of nodes in the hierarchical feature space is learned based on the Euclidean distance between nodes; the final attention is the result of the edge convolution operation activated by the sigmoid function, and the formula is as follows: Function Ψ represents the function applied to each level feature F (m) The operator N represents the edge convolution operation applied at each level, and It represents the merging of features at all levels, the symbol ‖ represents the marginal convolution operation on the merged features, and the sigmoid function σ is used to map the features to the range between 0 and 1, thereby indicating the importance of features at each level; finally, the attention map is weighted and added to the feature map obtained by the convolution output to generate the final output feature map.

5. The skeletal motion recognition method based on spatiotemporal topology learning according to claim 1, wherein: Step S4 designs time dynamic sequence pooling, constructs a time series feature dynamic matrix through the Koopman theory, and extracts high-order dynamic information in the time dimension. Specifically: The nonlinear system is analyzed by linear transformation in high-dimensional space through the Koopman operator; the Koopman operator is approximated in a computationally efficient form using dynamic mode decomposition; each eigenvector x is first embedded in the function ψ t Embedded in a higher-dimensional space, the transition between states is approximated by a linear equation, which is formulated as follows: x t+1 =(Kψ)(x) t ) Where K represents the Koopman operator; a linear approximation matrix D is constructed using dynamic mode decomposition, which controls the temporal dynamics of the sequence; given the characteristic matrix X of consecutive time steps 1:m-1 and X 2:m , the transfer matrix formula inspired by the Koopman operator is: Among them, X 1:m-1 is the feature matrix of time steps 1 to m-1, X 2:m is the feature matrix from time step 2 to m, Represents the pseudo-inverse matrix; the matrix D captures the basic dynamic behavior of the sequence and is used as a pooled representation of the temporal structure; for classification, each action category is assigned a learnable matrix D that encodes its unique temporal features. For a given sequence, D is used i ·X t Calculate the evolved state and compare it to the next state X by Euclidean distance q For comparison, the formula is as follows: s i =-d i s i It is the inverse of the distance and is regarded as the classification score. The final prediction of all class scores is made through SoftMax: Among them, p i is the probability of the ith class, and N is the total number of classes.

Images