Lightweight action recognition method based on multi-scale separation topology network

Abstract

The invention discloses a lightweight action recognition method and system based on a multi-scale separation topology network, and belongs to the technical field of computer vision and artificial intelligence. The technical problem to be solved by the invention is to overcome the defects that an original space-time diagram convolutional network is large in parameter quantity, high in calculation redundancy and difficult to deploy in real time in a clinical rehabilitation training environment with limited calculation resources. According to the technical scheme, the method is characterized in that a network architecture comprising a feature preprocessing layer and a plurality of cascaded lightweight graph convolution blocks is constructed; wherein the feature preprocessing layer realizes channel expansion and time sequence dimension reduction through convolution; the space graph convolution module adopts a channel separation strategy and a learnable adjacent matrix mechanism, and adaptively learns an optimal connection relationship between joint points while reducing parameters; the time convolution module adopts a multi-scale design, and integrates pooling and deformable convolution branch so as to efficiently capture multi-granularity time sequence characteristics from fine joint movement to a complete action cycle. The method has the main beneficial effects that on the premise of ensuring the action recognition precision, the model volume is remarkably compressed, and the reasoning speed is greatly improved, so that the real-time deployment of a high-precision action quality evaluation system on embedded equipment becomes possible, and the urgent demand of clinical rehabilitation training on real-time feedback is met.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a deep learning-based method for human skeleton motion recognition. Specifically, it is a lightweight spatiotemporal graph convolutional network architecture optimized for rehabilitation training scenarios, which has high real-time performance and high computational efficiency. Background Technology

[0002] Human motion recognition is a core technology at the intersection of computer vision and artificial intelligence, with significant application value in many fields such as intelligent rehabilitation medicine, human-computer natural interaction, intelligent video surveillance, and sports science analysis. Traditional motion recognition methods are mainly based on RGB video frames or depth image sequences, extracting appearance and motion features through two-dimensional or three-dimensional convolutional neural networks. However, these methods are susceptible to interference from factors such as cluttered backgrounds, changes in lighting, differences in viewing angles, and occlusion, thus affecting the accuracy and robustness of the recognition results.

[0003] To overcome the aforementioned limitations, methods based on human skeletal keypoint data have received widespread attention in recent years. Skeletal data directly represents actions as a sequence of coordinates of human joints, possessing stronger semantic information and high-level abstraction capabilities, and exhibiting natural robustness to changes in scene appearance. Among numerous action recognition models based on skeletal data, the Spatial-Temporal Graph Convolutional Network (ST-GCN) is considered a representative work. This model models the human skeleton as a spatiotemporal graph structure, where nodes correspond to joints, and edges are constructed based on the natural connectivity and temporal continuity of the human body. It utilizes graph convolution operations to simultaneously learn the spatial dependencies between joints and the dynamic evolution patterns of actions in the temporal dimension.

[0004] While ST-GCN provides an effective paradigm for skeletal motion recognition, it and most of its derivative models still have several limitations that need to be addressed: First, it suffers from high computational complexity. Both its spatial graph convolution and temporal convolution modules have a large number of parameters and redundant calculations, resulting in a large overall model size that is difficult to adapt to real-world applications with limited computing resources. Second, its real-time performance is poor. The high computational load prevents the model's inference speed from meeting the real-time feedback requirements of systems such as rehabilitation training, limiting its feasibility for deployment on embedded devices or mobile terminals. Finally, its expressive power is limited. On the one hand, its spatial graph convolution usually relies on a predefined, fixed adjacency matrix of human physical structures, making it difficult to adaptively learn the implicit functional associations between joints that are highly related to specific movements. On the other hand, the temporal convolution structure it uses is mostly single-scale with a limited receptive field, making it difficult to simultaneously and effectively capture multi-granular temporal features, ranging from subtle joint vibrations to macroscopic movement cycles.

[0005] Therefore, in the face of the dual requirements of efficiency and performance in practical applications, there is an urgent need in this field for a novel lightweight action recognition network architecture that can significantly reduce the number of model parameters and computational overhead while maintaining high recognition accuracy, and enhance its ability to represent complex spatiotemporal features. Summary of the Invention

[0006] Purpose of the Invention: The purpose of this invention is to overcome the shortcomings of existing technologies and provide a lightweight motion recognition network architecture and method. This architecture achieves an optimal balance between model accuracy and efficiency through deep optimization of spatial and temporal modeling modules, making it particularly suitable for deployment in resource-constrained environments, providing real-time motion quality assessment for rehabilitation training.

[0007] The core concept of this invention lies in constructing a highly efficient and lightweight spatiotemporal graph convolutional network through four major technological innovations: feature preprocessing and dimensionality reduction, channel separation and grouped convolution, learnable adaptive graph structure, and multi-scale temporal feature fusion. This method significantly reduces the number of model parameters and computational overhead while maintaining or even improving the accuracy of action recognition. It aims to promote the real-time application of high-precision action quality assessment technology on embedded devices, meeting the urgent need for real-time feedback in clinical and home rehabilitation settings.

[0008] Technical solution: The lightweight action recognition network of the present invention is characterized by comprising, in sequence, a feature preprocessing layer, a spatial graph convolution module, and a temporal graph convolution module.

[0009] The feature preprocessing layer (Stem Layer), as the first stage of the network, plays a crucial role in performing preliminary feature enhancement and structural reshaping on the original input data. Specifically, this layer receives an input skeletal sequence tensor with dimensions (B, 3, T, V), where B is the batch size, 3 represents the three-dimensional spatial coordinates (x, y, z) of each joint, T is the number of time frames (typically 256 frames), and V is the number of joints (e.g., 25). The processing of this layer includes the following key steps: First, a linear transformation layer maps the low-dimensional spatial coordinates of each joint to a high-dimensional feature space, expanding the number of channels from 3D to 64D, greatly enhancing the model's representational ability and enabling subsequent networks to learn and capture more complex and subtle movement patterns in this high-dimensional space. Second, learnable positional encoding parameters are introduced to generate unique positional feature vectors for each joint, enhancing the model's ability to perceive the human body's spatial topology. Subsequently, the features are re-dimensioned, merging the dimensions of the number of people, joints, and feature channels, and applying one-dimensional batch normalization to stabilize the data distribution and accelerate training convergence. Finally, through temporal compression, the sequence length T is compressed from 256 frames to 64 frames. This operation is not a simple downsampling, but rather integrates the temporal information of multiple frames while reducing dimensionality, effectively capturing the initial dynamic features of the action, and significantly reducing the computational load and memory usage of all subsequent computational layers in the first step.

[0010] After processing by the feature preprocessing layer, the output data dimension becomes (B, 64, 64, V). This design achieves a balance between computational efficiency and feature richness in the initial stage of the network. The output of the feature preprocessing layer is then used as input, sequentially passed through four cascaded, structurally identical lightweight graph convolutional blocks. Each block is the core innovative unit of this invention, responsible for progressively extracting high-level semantic features in both spatial and temporal dimensions. Each block consists of a Channel-Separate Topology Adaptive GCN (CSTA-GCN) and a Multi-scale Deformable Temporal Convolutional CN (MD-TCN) connected in series.

[0011] The Channel-Separate Topology Adaptive Spatial Graph Convolutional Module (CSTA-GCN) is a lightweight and enhanced improvement of traditional spatial graph convolution. Its features include: (1) Channel-Separate Design: The module first evenly splits the input feature map into S groups (S is an integer greater than 1, e.g., S = 8) along the channel dimension. For each subgroup of features, a graph convolution operation is performed independently. This strategy decomposes a heavy full-channel graph convolution computation into multiple lightweight, parallel subgraph convolution operations. By reducing the number of parameters in the convolution kernel, the model complexity and computational overhead are significantly reduced, while allowing the network to learn diverse spatial patterns in different feature subspaces, enhancing the model's expressive power. Finally, the outputs of all subgroups are concatenated along the channel dimension to restore the complete number of channels, achieving preliminary information fusion.

[0012] (2) Learnable Adjacency Matrix: To overcome the limitations of predefined human physical connection topologies, the module introduces a trainable parameter matrix with the same dimensions as the adjacency matrix. This matrix is ​​optimized during training using gradient descent and L2 regularization is applied to prevent overfitting. This mechanism enables the model to adaptively learn the optimal functional connectivity relationships between joints that are highly relevant to the action recognition task from the data, overcoming the limitations of predefined human physical connection topologies and greatly enhancing the flexibility and accuracy of spatial modeling.

[0013] (3) Topological Distance Encoding: Based on the learnable adjacency matrix, this invention further incorporates an untrainable topological bias matrix based on biomechanical prior knowledge, which, when added to the learnable adjacency matrix, jointly guides spatial feature aggregation. This matrix is ​​generated by calculating the shortest path distance between key points on a predefined skeleton map, enhancing the interpretability and stability of spatial relationships and forming a complete topological adaptive mechanism. This ensures that the model, while possessing strong learning capabilities, avoids learning connection relationships that violate physical laws.

[0014] Multi-scale Deformable Temporal Convolution (MD-TCN) employs an innovative multi-scale temporal modeling scheme. This module, which follows the spatial module, is responsible for performing multi-granularity temporal dynamic modeling on spatially aggregated feature sequences. Its design incorporates two core innovations:

[0015] (1) Multi-Branch Architecture: The module adopts an 8-branch structure, each focusing on feature extraction at different time scales. The first branch uses a one-dimensional standard convolution with a kernel size of 3, focusing on capturing the instantaneous subtle changes in joint motion and high-frequency dynamic patterns; the second branch uses max pooling to extract the salient contextual information of features at zero parameter cost; the third branch uses average pooling to provide smooth contextual features; the remaining five branches are deformable convolution branches, using one-dimensional dilated convolutions with dilation rates of 1, 2, 3, 4, and 5, respectively, to capture dynamic patterns ranging from medium-range to long-term temporal dependencies through different receptive fields.

[0016] (2) Grouped convolution and feature fusion: Each branch adopts a grouped convolution strategy to divide the feature channels into 4 groups for independent processing, which reduces the number of parameters of this module by 75%. The outputs of all branches are concatenated in the channel dimension and cross-channel information fusion and dimensional unification are achieved through a 1×1 convolutional layer to generate the final feature output that integrates multi-scale temporal information.

[0017] Compared with existing technologies, this invention has the following significant advantages: (1) Extremely high computational efficiency: Through a series of methods such as channel separation, grouped convolution, and multi-branch lightweight design, this invention significantly reduces the number of model parameters and floating-point operations (FLOPs), and significantly improves inference speed, which can meet the stringent real-time requirements of rehabilitation training systems. (2) Powerful performance: The introduction of learnable adjacency matrix and topological coding enhances the flexibility of spatial modeling; multi-scale temporal convolution ensures the effective extraction of temporal features at different granularities. These innovations enable the lightweight model to match or even surpass larger and heavier traditional models in terms of accuracy. (3) Excellent deployment adaptability: The lightweight model size allows it to be easily deployed on edge devices with limited computing resources, reducing the threshold and cost of clinical applications. Attached Figure Description

[0018] Figure 1 A flowchart of a method according to an embodiment of the present invention;

[0019] Figure 2 Detailed structural diagram of a single lightweight graph convolutional block (LW-GCN Block) in this invention;

[0020] Figure 3 Internal structure diagram of the channel separation topology adaptive spatial graph convolution module (CSTA-GCN Module) of this invention; Detailed Implementation

[0021] The present invention provides a lightweight action recognition method based on depth-separable spatiotemporal graph convolution, which can be implemented through the following embodiments. To verify the scalability of the algorithm, this embodiment uses the large-scale publicly available NTU RGB+D 60 skeletal action recognition dataset as the implementation object. This dataset contains 60 categories of human actions, performed by 40 volunteers, and provides accurate 3D skeletal keypoint data.

[0022] First, data preprocessing and input configuration are performed. Skeletal sequence data is read from the dataset. The original single-segment skeleton data is represented as a four-dimensional tensor, with dimensions of batch size, number of time frames, number of joints, and coordinate dimension. Unevenly long sequences are unified to 256 frames using interpolation, with a fixed number of 25 joints. Each joint contains three spatial coordinate dimensions: x, y, and z. The batch size is set to 32 for training and 64 for validation. The data tensor dimensions are adjusted to (batch size, number of channels, number of time frames, number of joints), i.e., (32, 3, 256, 25), as the network input.

[0023] Next, a feature preprocessing layer performs preliminary feature transformation on the input data. This layer consists of a one-dimensional convolutional layer with a kernel size of 1, a stride of 4 in the time dimension, a stride of 1 in the key dimension, and the number of output channels expanded to 64. No bias term is used, and a LeakyReLU activation function is applied after the convolution operation. After this layer, the input data dimension is transformed from (32,3,256,25) to (32,64,64,25), achieving feature expansion in the channel dimension and dimensionality reduction in the time dimension, significantly reducing subsequent computational costs.

[0024] The preprocessed feature tensors are then sequentially input into four cascaded lightweight graph convolutional blocks for spatiotemporal feature extraction. Each lightweight graph convolutional block contains a spatial graph convolutional module and a temporal convolutional module. The spatial graph convolutional module employs a channel separation strategy, uniformly dividing the input channels into four groups of 16 channels each, and performing independent graph convolution operations on each group. The graph convolution operation is implemented based on a learnable adjacency matrix and a fixed topological bias matrix, where the topological bias matrix is ​​generated based on the shortest path distance between keypoints. After completing the graph convolution operation on each subgroup of features, the outputs are concatenated along the channel dimension, and cross-channel information fusion is achieved through pointwise convolution, resulting in a feature tensor with the same dimension as the input.

[0025] The temporal convolution module employs a multi-branch parallel structure with a total of 8 independent branches. Branch 1 uses a standard 1D convolution with a kernel size of 3; branches 2 and 3 use 1D max pooling and average pooling operations with kernel sizes of 3, respectively; branches 4 through 8 use 1D dilated convolutions with kernel sizes of 3, with dilation rates set to 1, 2, 3, 4, and 5, respectively. All branches internally employ a grouped convolution strategy, with 4 groups. After concatenating the outputs of each branch along the channel dimension, a 1x1 convolutional layer is used to achieve cross-channel information fusion and dimensionality reduction, resulting in a feature tensor with the output dimension remaining unchanged.

[0026] After processing through four cascaded lightweight graph convolutional blocks, global average pooling is performed on the resulting high-level feature tensor in both the temporal and spatial dimensions, compressing it into a one-dimensional feature vector. This feature vector is then input into a fully connected classification layer, whose output unit number corresponds to the 60 action categories in the dataset. Finally, the predicted probability distribution for each category is obtained through the Softmax function.

[0027] During model training, the standard cross-entropy loss function was used as the optimization objective, and the Adam optimizer was used for parameter optimization. The initial learning rate was set to 0.05, and a cosine annealing scheduling strategy was employed for learning rate decay. Training lasted for 140 epochs, with a batch size of 64. To improve the model's generalization ability, L2 regularization constraints were applied to the learnable adjacency matrix, with a weight decay coefficient set to 1e-4. Dropout was applied after each convolutional layer, with a dropout rate of 0.1.

[0028] The lightweight network provided in this embodiment achieved excellent performance on the NTU RGB+D 60 dataset. It maintained high recognition accuracy while significantly reducing the number of parameters and computational cost, indicating that the network in this paper can also be extended to other action recognition tasks, fully meeting the needs of real-time action recognition and evaluation on resource-constrained embedded devices.

Claims

1. A method for lightweight action recognition based on multi-scale separated topological network, characterized in that, Comprising the following steps: Step (1): data collection and processing, using a camera to collect 8 common rehabilitation actions (alternating palm turning of upper limbs, alternating touching of shoulders with both hands, alternating touching of lumbar vertebrae with both hands, alternating finger-to-nose with upper limbs, standing for 30 seconds, 10m walking back and forth, going up and down 3 levels of stairs, standing and sitting), 20 subjects are divided into training group and test group according to the cross-subject division standard, to ensure that the subjects of the training set and the test set are disjoint, and this division method can effectively evaluate the generalization ability of the model on subjects that have not been seen before; Step (2): data preprocessing and feature embedding, the dimension of the read skeletal sequence data is uniformly adjusted to (B, C, T, V), where B is the batch size, C = 3 represents the three-dimensional space coordinates, T = 256 is the number of time frames unified by the interpolation method, and V = 25 is the number of joint nodes. Then, the following preprocessing and feature enhancement operations are performed in turn, feature mapping and position encoding are added, then dimension rearrangement and standardization are performed, and finally time compression is performed. Step (3): input the preprocessed data into the channel separation topological adaptive spatial graph convolution module, adopt the channel separation strategy to uniformly divide the C input channels into 8 groups, each group has C / 8 channels, and the graph convolution operation is independently performed, and a learnable adjacency matrix with a dimension of 25*25 is used, which is optimized through gradient descent during training. At the same time, a fixed topological bias matrix based on the shortest path distance between joint nodes is added, which guides the spatial feature aggregation together; Step (4): input the spatial features into the multi-scale deformable temporal convolution module, adopt an 8-parallel branch structure, including a standard convolution branch, a max pooling branch, an average pooling branch, and five deformable convolution branches with expansion rates of 1-5, respectively. Each branch uses a grouped convolution strategy inside, with 4 groups, and the outputs of each branch are concatenated in the channel dimension and then fused through a 1*1 convolution; Step (5): compress the spatio-temporal features through global average pooling and input them into the fully connected layer, the input dimension of the fully connected layer is 64, and the output dimension is 8, corresponding to the 8 action categories of the dataset, and finally the class probability distribution is output through the Softmax function.

2. The method of claim 1, wherein, In step (1), the cross-subject division standard is used to divide 10 of the 20 subjects into the training group and the other 10 into the test group. This division method can ensure that different subjects are faced during the training and testing of the model, thereby better evaluating the generalization performance of the model.

3. The method of claim 1, wherein, The data preprocessing and feature embedding in step (2) comprises: using an interpolation method to unify original bone sequences of different lengths to T frames, maintaining consistency of the time dimension; mapping low-dimensional spatial coordinates of each joint to a high-dimensional feature space through a linear transformation layer, enhancing feature representation capability; introducing learnable position encoding parameters to generate unique position feature vectors for each joint, enhancing the model's perception of the spatial topology of the human body; dimensionally reorganizing the features, combining the number of people, joint and feature channel dimensions, and applying one-dimensional batch normalization processing to stabilize data distribution and speed up training convergence; reducing and integrating the time dimension to better capture fine-grained temporal features and enable the model to learn more rich features.

4. The method of claim 1, wherein, The spatial graph convolution module in step (3) adopts a core design combining channel separation strategy and learnable topology, specifically comprising: a channel separation operation uniformly divides C channels of input features into 8 groups, each containing C / 8 channels, and performs independent graph convolution operations, which significantly reduces model complexity and computational overhead by reducing the parameter amount of convolution kernels, and is the key to realizing network lightweight; a learnable adjacency matrix mechanism is adopted, which is initialized as an identity matrix at the beginning of training and introduces L2 regularization constraints to prevent overfitting, enabling the model to adaptively learn the optimal functional connection relationship between joints and the action recognition task from data; a topological distance encoding is introduced, which generates a fixed topological bias matrix by calculating the shortest path distance of the joints on the human skeleton graph, and the matrix is added to the learnable adjacency matrix to jointly guide spatial feature aggregation, which provides the model with prior knowledge of human biomechanics, enhances the explainability and stability of spatial relationships, and forms a complete topological adaptive mechanism.

5. The method of claim 1, wherein, The time convolution module in step (4) adopts a multi-branch parallel structure to efficiently capture multi-granularity temporal dynamic features, specifically comprising: 8 parallel branches respectively focusing on feature extraction of different time scales: one standard convolution branch is used to capture local fine temporal patterns; the maximum pooling and average pooling branches extract the context information of the features with zero parameter cost; five deformable convolution branches with different dilation rates (e.g. 1, 2, 3, 4, 5) capture different scale dynamic patterns from medium-range to long-term temporal dependencies; each branch internally adopts a grouped convolution strategy, which divides the feature channels into 4 groups for independent processing, reducing the parameter amount of the module by 75%, while ensuring multi-scale perception ability and greatly improving the parameter efficiency of the model; the outputs of all branches are concatenated in the channel dimension, and a 1x1 convolution layer is used to realize cross-channel information fusion and dimension unification, generating a final feature output that integrates multi-scale temporal information.

6. The method of claim 1, wherein, The fully connected layer in step (5) receives a 64-dimensional feature vector compressed by global average pooling and outputs an 8-dimensional feature vector corresponding to the logits of 8 action categories, which is finally converted into a probability distribution by the Softmax function to complete the action classification task.

Images