A skeleton point gesture recognition method based on graph convolution and star attention
Patent Information
- Application Number
- CN202610635471.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-09
- Publication Date
- 2026-08-18
AI Technical Summary
[0005]然而，现有的ResNet50分类模型存在两大局限：(1)参数量23.6M、模型体积90MB，难以满足边缘设备轻量化需求；(2)未显式利用手部骨骼拓扑约束，对关键点长程依赖建模不足，且时空特征融合粗糙，限制了双手复杂动态交互的判别能力
本方案融合Graph Transformer的全局注意力与StarNet的星操作机制，构建高效时空建模架构，空间维度采用图卷积与星注意力联合建模局部拓扑与全局依赖；时间维度通过多尺度卷积与时序注意力捕捉动态特征；并引入连续时空交替机制实现时空信息的深度耦合。此外，设计双手几何特征辅助路径增强双人交互判别能力，在mmVR数据集上的实验表明，GSSTANet达到94.22%的识别准确率，参数量仅0.37M，模型大小1.43MB，在保持高精度的同时较ResNet50压缩63倍，为资源受限场景下的雷达骨骼手势识别提供了高效解决方案。
Smart Images
Figure CN122598256A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a gesture recognition system in VR / AR interaction, specifically a skeletal point gesture recognition method based on graph convolution and star attention. Background Technology
[0002] Gesture recognition is a core technology for immersive VR / AR interaction. While traditional methods such as vision, wearable sensors, WiFi, and ultrasound have high recognition rates, they are easily limited by changes in lighting and privacy concerns. Millimeter-wave radar, due to its insensitivity to light, high resolution, and privacy protection features, has become an ideal alternative.
[0003] Deep learning has driven research into millimeter-wave radar gesture recognition, but existing methods are mostly based on coarse-grained spectra or point clouds, making it difficult to characterize the fine movements and anatomical structures of the hand. To meet the refined requirements of VR / AR for natural interaction, research has shifted to fine-grained skeletal gesture recognition—first estimating key points of the hand skeleton from radar point clouds, and then classifying based on the skeleton sequence. Skeletal-level representation can explicitly model the topology and joint constraints of the hand, offering significant advantages in interpretability and detailed modeling.
[0004] The existing technology combines multi-frame stitching, Transformer and PointNet++ to reduce the pose estimation error to 0.59cm and achieve head-mounted egocentric pose estimation, breaking through the limitations of fixed infrastructure. On this basis, it provides pre-estimated skeletal key points and mmVR dataset, enabling downstream gesture recognition accuracy to reach 90.8%.
[0005] However, the existing ResNet50 classification model has two major limitations: (1) it has 23.6M parameters and a model size of 90MB, which is difficult to meet the lightweight requirements of edge devices; (2) it does not explicitly utilize the topological constraints of the hand skeleton, has insufficient modeling of long-range dependencies of key points, and has coarse spatiotemporal feature fusion, which limits the ability to distinguish complex dynamic interactions of the hands. Summary of the Invention
[0006] This invention provides a skeletal gesture recognition method based on graph convolution and star attention. It is based on the GSSTANet skeletal gesture recognition network based on graph convolution and star attention, and achieves efficient space-time joint modeling through star attention mechanism and continuous spatiotemporal alternation modeling.
[0007] The present invention adopts the following technical solution: A skeletal point gesture recognition method based on graph convolution and star attention includes an input embedding module, a continuous spatiotemporal alternation module, and a DHGF lightweight auxiliary module.
[0008] The input embedding module is based on the actual number of key points. Dynamically constructing an undirected adjacency matrix The single-hand topology is based on the standard 21-key hand skeleton, where node 0 is the root node of the wrist / palm, directly connected to the root of the five fingers (nodes 1, 5, 9, 13, 17), and each finger is connected sequentially along the joint chain (thumb: 0→1→2→3→4; the other fingers are similar); the two-hand input adopts a block diagonal topology: the left and right hands (nodes 0–20 and 21–42) are independently constructed as single-hand connections, without crossing hand edges; The model input is a sequence of hand key points. ,in For batch size, For the number of frames in the sequence, Indicates one hand or two hands. The number of key points per hand, with the last dimension representing three-dimensional coordinates. ; To achieve unified processing for single / two-handed input, if the input is single-handed ( Then, inserting a zero pseudo-hand in the two-hand dimension yields... Meanwhile, the original coordinates of both hands are preserved (for subsequent calculation of the geometric features of both hands), and for input from both hands, they are merged along the keypoint dimension: Among them, the 21-dimensional fill is zero when using one hand, and when using two hands... Subsequently, the coordinates are mapped to the feature dimension using point-wise linear projection. : ,in , LN represents LayerNormalization, where L is the ReLU activation function.
[0009] The continuous spatiotemporal alternation module achieves the gradual refinement and mutual enhancement of spatial structure and temporal dynamics at each level by alternating between spatial and temporal updates. Spatial updates independently perform graph convolution and Star Attention operations on each frame; temporal updates independently model each key point along the time dimension. The graph convolution described above models the spatial adjacency relationships between joints, given input features. (in For the number of joints, (for feature dimensions), and a predefined adjacency matrix for the hand skeleton. Construct a normalized symmetric adjacency matrix: ,in To form the degree matrix, self-loops are added to enhance the weights of the node's own features; the basic form of spatial graph convolution is: ,in For learnable weights, It is a non-linear activation function.
[0010] The Star Attention mechanism introduces a global summary token (called the star token) and inputs it together with local node features into the multi-head self-attention mechanism, thereby achieving efficient global information interaction. Given the node features after convolution of a graph First, calculate the global star token: The star token is then concatenated into the original sequence to form an extended sequence. Multi-head self-attention (MHA) is applied to this sequence: , The attention function adopts the standard scaled dot-product form, and the attention output is separated into the updated star token after residual connection and layer normalization. To further enhance the nonlinear expressive power of features, this invention introduces StarOperation after attention. This operation generates features through two independent linear paths and achieves efficient high-order interactions in the form of element-wise multiplication. , ,in It is the ReLU6 activation function. This is for subsequent linear transformations and layer normalization.
[0011] The specific execution logic of the continuous spatiotemporal alternation module is as follows: Suppose the network contains Alternation (i.e.) Each spatial layer and (if the time layer is paired), then the first time layer pairing) The alternating inputs are denoted as (in , , , (These represent batch size, number of frames, number of joints, and feature dimension, respectively). Spatial updates perform graph convolution and Star Attention operations independently for each frame: ,in Indicates the first A spatial processing layer, This is the normalized adjacency matrix of the hand skeleton; Time updates, on the other hand, model each key point independently along the time dimension:
[0012] in Indicates the first A time processing layer.
[0013] Time layer A combined structure of "multi-scale local dynamics + long-range dependency + Star Operation enhancement" is adopted; time series features of individual key points are analyzed. (Rearranged as) First, multi-scale one-dimensional depthwise separable convolutions (kernel sizes) are applied in parallel. Capturing local dynamics at different time scales:
[0014] The multi-scale features are then concatenated along the channel dimension and fused using a 1×1 convolution:
[0015] The DHGF lightweight auxiliary module explicitly captures the dynamic relationship between key points of both hands without interfering with the spatiotemporal modeling of the backbone network. DHGF first extracts the coordinates of key points of the left and right hands from the skeleton sequence, calculates their Euclidean distance and first-order difference in the time dimension (velocity), flattens the distance and velocity features into low-dimensional vectors, and obtains a compact representation of hand interaction through 2-3 layers of MLP (ReLU+BN) nonlinear mapping. This vector is then concatenated and fused with the classification head after the backbone feature extraction, providing exclusive discrimination clues for hand movements, while having little impact on single-person gestures.
[0016] The present invention has the following advantages over the prior art: This solution integrates the global attention of Graph Transformer and the star operation mechanism of StarNet to construct an efficient spatiotemporal modeling architecture. In the spatial dimension, graph convolution and star attention are used to jointly model local topology and global dependencies; in the temporal dimension, multi-scale convolution and temporal attention are used to capture dynamic features; and a continuous spatiotemporal alternation mechanism is introduced to achieve deep coupling of spatiotemporal information. In addition, a geometric feature-assisted path for both hands is designed to enhance the ability to distinguish between two people's interactions. Experiments on the mmVR dataset show that GSSTANet achieves a recognition accuracy of 94.22%, with only 0.37M parameters and a model size of 1.43MB. While maintaining high accuracy, it is 63 times more compressed than ResNet50, providing an efficient solution for radar skeletal gesture recognition in resource-constrained scenarios. Attached Figure Description
[0017] Figure 1 This is a schematic diagram of the overall model of the present invention.
[0018] Figure 2 A schematic diagram of the 21 standard key points of the hand bones.
[0019] Figure 3 This is a schematic diagram of Star Attention.
[0020] Figure 4 This is a schematic diagram of the skeletons for 8 different hand gestures.
[0021] Figure 5 A confusion matrix plot drawn on the test set. Detailed Implementation
[0022] The technical solution of the present invention will be demonstrated and verified in conjunction with experiments below.
[0023] like Figure 1 As shown, this invention designs a lightweight continuous spatiotemporal alternation network. The network model receives the key point sequence of the 3D skeleton of the hand as input. The architecture design is compatible with single-handed and two-handed input scenarios. It mainly consists of an input embedding layer, a continuous spatiotemporal alternation backbone (stacked spatial-temporal processing layer), a two-handed geometric auxiliary path (DHGF), and a classification head.
[0024] Core innovations include: (1) Star Attention mechanism, which achieves efficient global context aggregation and non-linear enhancement through star token and lightweight Star Operation; (2) A continuous spatiotemporal alternation mechanism to achieve a layer-by-layer refined interaction of spatial and temporal characteristics; (3) The task-specific DHGF auxiliary path explicitly captures the geometric dynamics of the two-hand swipe interaction. The model has only 0.37M parameters and a model size of 1.43MB, achieving a maximum accuracy of 94.22% while maintaining extremely low computational overhead.
[0025] 1. Input and Processing The model input is a sequence of hand key points. ,in For batch size, For the number of frames in the sequence, Indicates one hand or two hands. The number of key points per hand, with the last dimension representing three-dimensional coordinates. .
[0026] To achieve unified processing for single / two-handed input, if the input is single-handed ( If a zero pseudo-hand is inserted in the two-hand dimension, then we get... At the same time, the original coordinates of both hands are preserved (for subsequent calculation of the geometric features of both hands). For input from both hands, the coordinates are merged along the keypoint dimension:
[0027] In the case of one hand, the 21 dimensions are all zero-filled; in the case of two hands... Subsequently, the coordinates are mapped to the feature dimension using point-wise linear projection. :
[0028] in , is the ReLU activation function, and LN represents Layer Normalization. This embedding process preserves the original spatial information of the skeletal keypoints while improving feature representation capabilities.
[0029] To incorporate prior knowledge of hand anatomy, this invention is based on the actual number of key points. Dynamically constructing an undirected adjacency matrix The single-hand topology is based on the standard 21 keypoint hand skeleton (as shown in Figure 2), where node 0 is the wrist / palm root node, directly connected to the root of the five fingers (nodes 1, 5, 9, 13, 17), and each finger is connected sequentially along the joint chain (thumb: 0→1→2→3→4; the other fingers are similar).
[0030] For biphasic input, a block diagonal topology is adopted: the left and right hands (nodes 0–20 and 21–42) are independently constructed as single-hand connections, with no cross-hand edges. This design preserves the local structural constraints within both hands while avoiding the introduction of unnecessary cross-hand interference, providing a reliable anatomical prior for subsequent graph convolutional layers.
[0031] 2. Spatial Modeling of Graph Convolution and Global Attention To achieve efficient spatial structure modeling, a local-global complementary spatial module is adopted, combining a graph convolutional network (GCN) to capture the local topology of hand anatomy with global context modeling based on star attention. This design follows a three-level progressive spatial modeling approach: "local graph structure modeling → global star attention → high-dimensional nonlinear enhancement," and is specifically optimized for the task of key point sequences of hand bones, achieving significant results under lightweight constraints.
[0032] (1) Graph convolutional layer with local topological capture To achieve efficient spatial structure modeling, considering the natural graph structure characteristics of the hand skeleton, a graph convolutional network (GCN) is first used to model the spatial adjacency relationships between joints, given the input features. (in For the number of joints, (for feature dimensions), and a predefined adjacency matrix for the hand skeleton. Construct a normalized symmetric adjacency matrix:
[0033] in To define the degree matrix, self-loops are added to enhance the weights of the node's own features. The basic form of spatial graph convolution is:
[0034] in For learnable weights, This is a non-linear activation function. This layer can effectively aggregate first-order neighborhood information and capture local dependencies between hand joints. Unlike traditional CNNs that assume grid data, graph convolution directly passes messages between hand bones on a non-Euclidean structure, which is more in line with the physiological constraints of hand movements. Furthermore, it has low computational cost and strong parameter sharing, making it very suitable for lightweight model design.
[0035] (2) Design motivation and implementation of star attention mechanism To further capture global long-range dependencies between joints while maintaining a lightweight design, and inspired by StarNet's simplification of attention computation in image recognition, a method was designed as follows: Figure 3 The StarAttention mechanism shown here achieves efficient global information interaction by introducing a global summary token (called the star token) and inputting it along with local node features into multi-head self-attention.
[0036] Given the node features after convolution of a graph First, calculate the global star token:
[0037] The star token is then concatenated to the original sequence to form an extended sequence. .
[0038] Apply multi-head self-attention (MHA) to this sequence:
[0039]
[0040] The attention function adopts the standard scaled dot-product form. After the attention output is processed through residual connections and layer normalization, the updated star token is extracted.
[0041] To further enhance the non-linear expressive power of features, a Star Operation is introduced after attention. This operation generates features through two independent linear paths and achieves efficient high-order interactions in the form of element-wise multiplication:
[0042]
[0043] in It is the ReLU6 activation function. This facilitates subsequent linear transformations and layer normalization. Compared to traditional FFN, this design significantly improves the nonlinear modeling capability of high-dimensional features while maintaining a lower parameter count.
[0044] Finally, the updated star token is broadcast back to all nodes as global information injection, achieving an organic fusion of local structure (GCN) and global context (Star Attention). Particularly noteworthy is the integration of this star-shaped attention and star operations with the graph convolution depth of the hand skeleton: graph convolution first structurally models local physiological connections, then star attention further supplements the global context while preserving the local structure, and finally, star operations enhance the nonlinear expression. This three-level progressive spatial modeling approach—"local graph structure modeling → global star attention → high-dimensional nonlinear enhancement"—is a targeted design for the specific task of hand skeleton keypoint sequences, achieving good results under lightweight constraints.
[0045] 3. Spacetime Alternation Mechanism Traditional gesture recognition methods typically process spatial structure (the relative configuration of all keypoints at the same moment) and temporal dynamics (the trajectory of the same keypoint in the sequence) separately, fusing them through simple concatenation or later feature stitching. This approach often leads to insufficient alignment of spatial and temporal information in deep feature representations, thus limiting the model's ability to capture complex spatiotemporal interactions. Inspired by iterative modeling paradigms, this invention employs a continuous spatiotemporal alternation module. Through alternating spatial and temporal updates, it achieves gradual refinement and mutual enhancement of spatial structure and temporal dynamics at each level. Assume the network contains... Alternation (i.e.) Each spatial layer and (if the time layer is paired), then the first time layer pairing) The alternating inputs are denoted as (in , , , These represent batch size, number of frames, number of joints, and feature dimension, respectively.
[0046] Spatial updates perform graph convolution and Star Attention operations independently for each frame:
[0047] in Indicates the first The mechanism employs a time-processing layer. This alternating mechanism deeply couples spatial topological information with temporal dynamic information at each layer. Compared to traditional one-time fusion strategies, it more naturally captures highly coupled spatiotemporal dependencies such as gestures, making it particularly suitable for complex gesture recognition tasks that simultaneously depend on pose topology and motion processes.
[0048] Time layer A combined structure of "multi-scale local dynamics + long-range dependency + Star Operation enhancement" is adopted. This is used for the time-series features of individual keypoints. (Rearranged as) First, multi-scale one-dimensional depthwise separable convolutions (kernel sizes) are applied in parallel. Capturing local dynamics at different time scales:
[0049] The multi-scale features are then concatenated along the channel dimension and fused using a 1×1 convolution:
[0050] This design can simultaneously model short-term rapid movements (such as rapid finger swings) and medium- to long-term pose transitions. After local dynamic modeling, temporal multi-head self-attention is further introduced to capture long-range dependencies. Finally, StarOperation is applied for high-order nonlinear feature enhancement, and residual connections, layer normalization, and dropout are used to ensure training stability and generalization ability when multiple layers are stacked.
[0051] 4. Geometric feature extraction of both hands To enhance the ability to discriminate two-handed interactive actions, a lightweight auxiliary module, DHGF, was designed to explicitly capture the dynamic relationship between key points of both hands without interfering with the spatiotemporal modeling of the backbone network. DHGF first extracts the coordinates of key points of the left and right hands from the skeletal sequence and calculates their Euclidean distance and first-order temporal difference (velocity). The distance and velocity features are flattened into low-dimensional vectors and then subjected to 2-3 layers of MLP (ReLU+BN) nonlinear mapping to obtain a compact representation of two-handed interactions. This vector is then concatenated and fused with the classification head after backbone feature extraction, providing specific discriminative cues for two-handed actions while having minimal impact on single-person gestures. The number of DHGF parameters is less than 5% of the total parameters. In ablation experiments, its independent introduction improved accuracy by 0.47%-1.13%, validating its effectiveness in improving the robustness of complex action recognition by explicitly modeling the spatiotemporal relationship of both hands while maintaining high efficiency.
[0052] 1. Dataset Description The mmVR dataset, based on the mmEgoHand project, is a large-scale dataset specifically designed for egocentric hand pose estimation and gesture recognition tasks using head-mounted millimeter-wave radar + IMU. It is currently the first known head-mounted millimeter-wave radar hand-related dataset that supports simultaneous single-hand / two-hand capture.
[0053] This dataset includes three main human postures: standing, sitting, and lying down. It features three different indoor scenarios, 10 volunteers, and considers combinations of factors such as left / right hand, dominant / non-dominant hand, height, and weight. The specific gestures for the eight gestures are as follows: Figure 4 As shown, each gesture and combination is repeated 20 times. Total sample size: 5206 complete gesture samples after processing. Each sample contains 30 frames, with 3D coordinates of 21 keypoints (single hand) or 42 keypoints (both hands) per frame.
[0054] The experiment was conducted on the Ubuntu 20.04 operating system, with an Intel Xeon Gold 6348 CPU, an NVIDIA Quadro RTX 6000 GPU (24 GB of video memory), and the Torch 2.5.1 + CUDA 12.4 deep learning framework.
[0055] This work focuses on lightweight optimization for downstream gesture classification tasks, with input consisting of a 30-frame sequence of 3D skeletal keypoints for the hand (21 points for one hand / 42 points for both hands). Pre-estimated skeletal keypoints from the publicly available mmVR dataset are used. This data (approximately 72.73 mm MPJPE) is inferred from millimeter-wave radar + IMU signals by the mmEgoHand pre-trained model, and is completely consistent with the data used in the original work (ResNet, 90.80%), ensuring baseline fairness and comparability. No additional processing is performed to maintain reproducibility, and the dataset is divided according to the official train_list / test_list.
[0056] Training configuration: AdamW optimizer, initial learning rate 5e-4 (halved when alternating blocks ≥ 3), cosine annealing scheduling; loss function is weighted cross-entropy (inverse frequency weights + biphasic gain), label smoothing 0.1; gradient clipping threshold 2.0. Training lasts 350 epochs, batch size 32. Evaluation metrics include overall accuracy and class-specific Precision / Recall / F1.
[0057] 2. Experimental Results (1) Confusion matrix analysis To evaluate the model's classification performance, a confusion matrix is plotted on the test set. Figure 5 The diagonal elements represent the correct classification rate for each category, while the off-diagonal elements reflect misclassification patterns. GSSTANet achieves an average accuracy of 94.22% across all eight gesture categories. Notably, categories 6 and 7 (primarily two-handed gestures) achieve exceptionally high accuracy rates of 100% and 98%, respectively, demonstrating the model's strong ability to recognize complex two-handed interaction scenarios. This excellent performance is attributed to the DHGF module and the continuous spatiotemporal alternation mechanism, which effectively captures global context and local topology. Except for category 3, which has an accuracy of 87%, all other categories exceed 90%. Category 3 is mainly confused with category 4 (4%), with a small amount of overlap with categories 0, 1, and 2 (2-3% each). Misclassification may stem from intra-class variations in finger configuration and movement trajectories, as well as kinematic similarity with adjacent categories. Overall, the confusion matrix shows a clear dominance of the diagonal elements and minimal leakage from the off-diagonal elements, indicating that the spatiotemporal features extracted by the model have good inter-class discriminative power. The excellent performance of the two-handed gestures further validates the significant advantages of star-operation-enhanced attention and iterative spatiotemporal coupling strategies in capturing local geometric priors and long-range dependencies.
[0058] (2) Ablation test To verify the effectiveness of the Star module, the temporal-space alternation (STA) mechanism, and the DHGF module, a systematic ablation experiment was conducted on the target dataset. The results are shown in Table 1. The accuracies of Star, STA, and DHGF used alone were 91.66%, 92.51%, and 92.13%, respectively, indicating that each component is effective, with STA showing the most significant improvement (+0.85%). Paired combinations further revealed the synergistic effect between components: Star+STA reached 93.83% (an improvement of 2.17% over Star), STA+DHGF reached 93.27%, while Star+DHGF was only 91.94%, indicating that the gain of DHGF is mainly reflected when combined with STA.
[0059] The complete model integrating the three components achieved a maximum accuracy of 94.22%, which is an improvement of 2.37%, 1.52%, and 1.90% respectively compared to using Star, STA, and DHGF individually. The ablation results fully validated the key roles of each component and their complementary synergy.
[0060] Table 1. Accuracy of Ablation Module
[0061] (3) Comparison with existing methods To comprehensively evaluate model performance, we compared it with mainstream spatiotemporal graph convolutional networks (SPCCs) on the same dataset, including ResNet, ST-GCN, CTRGCN, MSG3D, GTNet, and DGCNN. Consistent training / test splits, data preprocessing, and evaluation metrics (Accuracy, F1, Precision, Recall) were used to ensure fair comparison. Table 2 shows the performance metrics, parameter count, and model size for each model. The results show that the proposed model significantly outperforms the others in all metrics: accuracy 94.22%, F1 score 0.9403, Precision 0.9411, and Recall 0.9416. It improves performance by 3.13% compared to the second-best method, ST-GCN (91.09%), and by 3.52% compared to the original ResNet50 (90.7%). Compared with other dedicated spatiotemporal models, its advantages are also obvious - it outperforms MSG3D (90.05%) by 4.17%, CTRGCN (90.33%) by 3.89%, DGCNN (83.03%) by 11.19%, and GTNet (81.02%) by 13.2%.
[0062] More importantly, while achieving the highest performance, the model has only 0.37M parameters and a model size of 1.43MB, far lower than the comparison methods (ST-GCN: 1.23M / 4.13MB; ResNet50: 23.61M / 90.26MB). This fully demonstrates the model's overwhelming advantages in computational efficiency and deployment friendliness, making it particularly suitable for resource-constrained edge device scenarios.
[0063] Table 2 Comparison of Models .
Claims
1. A skeletal point gesture recognition method based on graph convolution and star attention, characterized in that: It includes an input embedding module, a continuous spatiotemporal alternation module, and a DHGF lightweight auxiliary module.
2. The skeletal point gesture recognition method based on graph convolution and star attention as described in claim 1, characterized in that: The input embedding module is based on the actual number of key points. Dynamically constructing an undirected adjacency matrix The single-hand topology is based on the standard 21-key hand skeleton, where node 0 is the root node of the wrist / palm, directly connected to the root of the five fingers (nodes 1, 5, 9, 13, 17), and each finger is connected sequentially along the joint chain (thumb: 0→1→2→3→4; the other fingers are similar); the two-hand input adopts a block diagonal topology: the left and right hands (nodes 0–20 and 21–42) are independently constructed as single-hand connections, without crossing hand edges; The model input is a sequence of hand key points. ,in For batch size, For the number of frames in the sequence, Indicates one hand or two hands. The number of key points per hand, with the last dimension representing three-dimensional coordinates. ; To achieve unified processing of single / two-handed input, if the input is single-handed ( Then, inserting a zero pseudo-hand in the two-hand dimension yields... Meanwhile, the original coordinates of both hands are preserved (for subsequent calculation of the geometric features of both hands), and for input from both hands, they are merged along the keypoint dimension: , In the case of one hand, the 21 dimensions are all zero-filled; in the case of two hands... Subsequently, the coordinates are mapped to the feature dimension using point-wise linear projection. : Among them, the 21 dimensions are all zero-filled when using one hand, and when using two hands Subsequently, the coordinates are mapped to the feature dimension using point-wise linear projection. : ;in , is the ReLU activation function, and LN represents Layer Normalization.
3. The skeletal point gesture recognition method based on graph convolution and star attention according to claim 2, characterized in that: The continuous spatiotemporal alternation module achieves gradual refinement and mutual enhancement of spatial structure and temporal dynamics at each level by alternating between spatial and temporal updates. Spatial updates independently perform graph convolution and StarAttention operations on each frame; temporal updates independently model each key point along the time dimension.
4. The skeletal point gesture recognition method based on graph convolution and star attention as described in claim 3, characterized in that: The graph convolution described above models the spatial adjacency relationships between joints, given input features. (where N is the number of joints and D is the feature dimension), and a predefined adjacency matrix for the hand skeleton. Construct a normalized symmetric adjacency matrix: ,in To form the degree matrix, self-loops are added to enhance the weights of the node's own features; the basic form of spatial graph convolution is: ,in For learnable weights, It is a non-linear activation function.
5. The skeletal point gesture recognition method based on graph convolution and star attention according to claim 4, characterized in that: The Star Attention mechanism introduces a global summary token (called the star token) and inputs it together with local node features into the multi-head self-attention mechanism, thereby achieving efficient global information interaction. Given the node features after convolution of a graph First, calculate the global star token: The star token is then concatenated into the original sequence to form an extended sequence. Multi-head self-attention (MHA) is applied to this sequence: , The attention function adopts the standard scaled dot-product form, and the attention output is separated into the updated star token after residual connection and layer normalization. To further enhance the nonlinear expressive power of features, this invention introduces a Star Operation after attention. This operation generates features through two independent linear paths and achieves efficient high-order interactions in the form of element-wise multiplication. , ,in It is the ReLU6 activation function. This is for subsequent linear transformations and layer normalization.
6. The skeletal point gesture recognition method based on graph convolution and star attention according to claim 5, characterized in that: The specific execution logic of the continuous spatiotemporal alternation module is as follows: Suppose the network contains Alternation (i.e.) Each spatial layer and (if the time layer is paired), then the first time layer pairing) The alternating inputs are denoted as (in , , , (These represent batch size, number of frames, number of joints, and feature dimension, respectively). Spatial updates perform graph convolution and Star Attention operations independently for each frame: ,in Indicates the first A spatial processing layer, This is the normalized adjacency matrix of the hand skeleton; Time updates, on the other hand, model each key point independently along the time dimension: ,in Indicates the first A time processing layer.
7. The skeletal point gesture recognition method based on graph convolution and star attention according to claim 6, characterized in that: Time layer A combined structure of "multi-scale local dynamics + long-range dependency + Star Operation enhancement" is adopted; for the time series features of a single key point. (Rearranged as) First, multi-scale one-dimensional depthwise separable convolutions (kernel sizes) are applied in parallel. Capturing local dynamics at different time scales: The multi-scale features are then concatenated along the channel dimension and fused using a 1×1 convolution. .
8. The skeletal point gesture recognition method based on graph convolution and star attention according to claim 7, characterized in that: The DHGF lightweight auxiliary module explicitly captures the dynamic relationship between key points of both hands without interfering with the spatiotemporal modeling of the backbone network. DHGF first extracts the coordinates of key points of the left and right hands from the skeleton sequence, calculates their Euclidean distance and first-order difference in the time dimension (velocity), flattens the distance and velocity features into low-dimensional vectors, and obtains a compact representation of hand interaction through 2-3 layers of MLP (ReLU+BN) nonlinear mapping. This vector is then concatenated and fused with the classification head after the backbone feature extraction, providing exclusive discrimination clues for hand movements, while having little impact on single-person gestures.