Skeleton gesture recognition method based on difference relation and channel attention

Abstract

A skeleton gesture recognition method based on differential relations and channel attention is proposed. First, a dynamic gesture skeleton sequence is acquired. Based on the frame-level adjacency matrix formed by an existing skeleton graph convolutional network, differential relation units and attention relation units are introduced. The differential relation unit constructs an inter-frame differential relation matrix based on the changes in skeleton features between adjacent time frames. The attention relation unit groups the skeleton features by channel and calculates the joint attention relation matrix within each channel group. Subsequently, the frame-level adjacency matrix, the inter-frame differential relation matrix, and the channel-grouped attention relation matrix are fused to obtain the final enhanced adjacency matrix, which is then embedded into a graph convolutional unit for feature aggregation. Finally, the gesture category is output after global average pooling, a fully connected layer, and Softmax. This invention enhances the expressive power of inter-frame motion changes and channel subspace joint relationships in dynamic gestures, improving the accuracy and robustness of skeleton gesture recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, specifically relating to a skeleton gesture recognition method based on differential relationships and channel attention. Background Technology

[0002] With the development of intelligent robots, unmanned equipment, human-machine collaborative systems, and intelligent terminal interaction technologies, gestures, as a natural, intuitive, low-noise, and non-contact interaction method, have significant application value in scenarios such as robot control, unmanned system command, emergency rescue, and complex environment perception. In scenarios with strong noise, long distances, covert communication, or where voice commands are inconvenient, dynamic gestures based on visual perception can express operational intentions without relying on voice communication. Therefore, higher demands are placed on the accuracy, robustness, and spatiotemporal relationship modeling capabilities of gesture recognition methods.

[0003] Existing gesture recognition methods typically utilize RGB images, depth images, or skeleton sequences as input. RGB and depth images can preserve rich appearance information, but are easily affected by factors such as lighting variations, background interference, occlusion, and viewpoint changes. In contrast, skeleton sequences describe hand structure and movement trajectories through the coordinates of key hand points, mitigating the influence of background texture and lighting noise, and expressing joint connections and dynamic changes in a compact form. Therefore, dynamic gesture recognition based on skeleton sequences has become an important technical approach in gesture recognition and skeleton motion understanding.

[0004] Graph convolutional networks can model skeleton keypoints as graph nodes and joint connections as graph edges, thereby achieving joint neighborhood feature aggregation in the spatial dimension. Combined with temporal convolution, the model can further learn the dynamic changes of the skeleton sequence over time. Existing spatiotemporal graph convolutional methods typically perform feature aggregation based on predefined skeleton topologies, learnable adjacency matrices, or adaptive adjacency matrices. Furthermore, temporally decoupled graph convolutional networks can improve adjacency matrix modeling capabilities through channel decoupling learning and temporal decoupling learning, enabling more flexible relationship representations across different channels or time frames.

[0005] However, in dynamic gesture recognition tasks, different types of gestures often exhibit strong spatial pose similarities. Key differences may lie in the motion direction between adjacent frames, local joint displacement trends, finger bending and extension processes, and collaborative changes between different finger joints. While existing skeleton graph-based convolution methods can leverage the physical connectivity of the skeleton and frame-level topological relationships for feature aggregation, they are insufficient in modeling the motion difference relationships between adjacent frames, making it difficult to directly characterize local dynamic changes in hand joints from rest to motion, from bending to extension, and sliding in different directions. Furthermore, the joint relationships expressed in different feature channels or different channel groups are not entirely consistent. Existing methods do not adequately utilize joint attention relationships under channel grouping conditions, which can limit the model's ability to express fine-grained joint collaborative relationships.

[0006] In summary, existing skeleton graph convolution-based dynamic gesture recognition methods still suffer from the following problems: First, insufficient explicit modeling of inter-frame motion changes leads to class confusion for fast, large-amplitude, or inter-category similar dynamic gestures; second, insufficient utilization of joint relationships in different channel subspaces results in inadequate expression of fine-grained joint collaboration information; and third, the lack of a unified fusion mechanism for existing frame-level topology, inter-frame motion relationships, and channel attention relationships leaves room for improvement in the recognition accuracy and robustness of the model in complex dynamic gesture scenarios. Therefore, a skeleton gesture recognition method that can fuse inter-frame difference relationships and channel attention relationships on the basis of existing skeleton graph convolutional networks is needed. Summary of the Invention

[0007] To overcome the shortcomings of existing skeleton graph convolution-based dynamic gesture recognition methods, such as insufficient representation of inter-frame motion changes, inadequate utilization of joint coordination relationships under channel grouping conditions, and the lack of a unified fusion mechanism between existing frame-level topology, inter-frame motion relationships, and channel attention relationships, which easily lead to category confusion for fast dynamic gestures, large-amplitude gestures, and gestures with similar poses between categories, thus affecting recognition accuracy and robustness in complex scenarios, this invention provides a skeleton gesture recognition method based on differential relationships and channel attention. This method introduces differential relationship units to represent motion changes between adjacent frames and attention relationship units to represent joint associations in channel subspaces, based on the frame-level adjacency matrix formed by existing skeleton graph convolutional networks. These two types of enhanced relationships are integrated into the graph convolutional adjacency matrix, enabling the model to more fully utilize the motion trends, local changes, and cross-joint coordination information of hand joints while maintaining the stability of the original skeleton sequence recognition process, thereby improving the accuracy and robustness of dynamic gesture recognition.

[0008] The technical solution adopted by this invention to solve its technical problem is: A skeleton gesture recognition method based on differential relations and channel attention includes the following steps: Step 1: Obtain the skeleton sequence of the dynamic gesture to be recognized, and represent the skeleton sequence as an input tensor; Step 2: Based on the skeleton sequence, obtain the first skeleton graph convolutional network in the existing skeleton graph convolutional network. Frame adjacency matrix and the first Frame adjacency matrix As a common enhancement object for Differential Relational Unit (DRU) and Attention Relational Unit (ARU); Step 3: Based on the skeleton feature differences between adjacent time frames, construct the difference relation matrix using Differential Relationship Units (DRUs). ; Step 4: Based on the skeleton features after channel grouping, construct the attention relation matrix using Attention Relationship Units (ARUs). ; Step 5: The first... Frame adjacency matrix The difference relation matrix and the attention relationship matrix By merging, the final result is obtained. Frame Enhanced Adjacency Matrix ; Step Six: The final... Frame Enhanced Adjacency Matrix The TCN-GCN unit is embedded, and the gesture classification results are obtained based on graph convolution feature extraction, global average pooling, fully connected layers, and Softmax.

[0009] The beneficial effects of this invention are mainly reflected in the following aspects: Addressing the problems of rapid local joint changes, similar poses between categories, and significant differences in channel subspace relationships in dynamic gestures, this invention further adds inter-frame difference relationships and channel grouping attention relationships to the existing frame-level adjacency construction. Compared with methods that simply rely on fixed skeleton topology, learnable prior topology, or existing frame-level topology, this invention can simultaneously supplement motion change information and fine-grained joint relationship information during the graph convolution feature aggregation stage, without changing the skeleton sequence input, backbone stacking structure, or classification output format. This makes it easy to embed into existing skeleton graph convolutional recognition networks and is suitable for scenarios requiring continuous skeleton sequence understanding, such as dynamic gesture recognition, human-computer interaction, and robot control. Attached Figure Description

[0010] Figure 1 This is a diagram showing the overall structure of a skeleton gesture recognition method based on differential relationships and channel attention.

[0011] Figure 2 This is a flowchart of the adjacency matrix construction process for CDL and TDL.

[0012] Figure 3 This is a flowchart of the calculation process for the Differential Relational Unit (DRU).

[0013] Figure 4 This is a flowchart of the computation of the Attention Relationship Unit (ARU).

[0014] Figure 5 This is a diagram of the differential-attention adjacency enhancement structure in a single TCN-GCN unit. Figure 6 This is a schematic diagram of enhanced TD-GCN network data flow and tensor dimensions.

[0015] Figure 7 yes Results of the fusion weight ablation experiment. Detailed Implementation

[0016] The present invention will now be further described with reference to the accompanying drawings.

[0017] Reference Figures 1-7 A skeleton gesture recognition method based on differential relationships and channel attention includes the following steps: Step 1: Obtain the skeleton sequence of the dynamic gesture to be recognized, and represent the skeleton sequence as an input tensor: (1); in, , Indicates batch size, Indicates the number of feature channels. Indicates the number of time frames. Indicates the number of joints in the skeleton; In this embodiment, the original input coordinate channel number is The number of time frames is The number of skeletal joints is This input format can uniformly describe the correspondence between different samples, different time frames, and different keypoint positions, facilitating subsequent graph convolution modeling in both spatial and temporal dimensions. It should be noted that the above... This indicates the number of channels for the original input skeleton coordinates; within subsequent TCN-GCN units, This represents the number of input feature channels for the current network layer, which can be increased by the number of network layers. , Expand to .

[0018] Step 2: Based on the skeleton sequence, obtain the first skeleton graph convolutional network in the existing skeleton graph convolutional network. Frame adjacency matrix ,in, Indicates the time frame index. The first Frame adjacency matrix As a common enhancement target of the Differential Relational Unit (DRU) and the Attention Relational Unit (ARU), the first Frame adjacency matrix The source of its construction is as follows Figure 2 As shown.

[0019] The process of step two is as follows: Step 2.1: Construct the basic adjacency matrix. Specifically, the first... Frame adjacency matrix It can be obtained jointly from a predefined skeleton adjacency matrix, a learnable prior adjacency matrix, channel topology residuals, and frame-level topology residuals. (Predefined skeleton adjacency matrix) and learnable prior adjacency matrix Construct the basic adjacency matrix: (2); in, ; Step 2.2: Introduce channel topology residuals into the basic adjacency matrix, and then... By fusing these matrices into the basic adjacency matrix, a channel decoupling adjacency matrix is ​​obtained: (3); in, , , The fusion coefficient represents the channel topology residual; Step 2.3: Introduce the first [unclear] into the channel decoupling adjacency matrix. The frame-level topology residual corresponding to the frame, will be the frame-level topology residual. The result is obtained by fusing the decoupled adjacency matrix of the channel. Frame adjacency matrix: (4); in, , The fusion coefficients represent the frame-level topology residuals. This invention constructs an adjacency matrix for the link output based on an existing frame-level adjacency matrix, and uses it as the basis for enhancing subsequent differential and attention relationships.

[0020] Step 3: Based on the skeleton feature differences between adjacent time frames, construct the difference relation matrix using Differential Relationship Units (DRUs). ,in, Indicates batch size, This represents the number of skeleton joints; the calculation process of the Differential Relationship Unit (DRU) is as follows: Figure 3As shown, by explicitly calculating the changes in skeleton features between adjacent frames, DRU can highlight dynamic differences such as finger bending, stretching, sliding, and local displacement, enabling the adjacency matrix to acquire motion perception capabilities.

[0021] The process of step three is as follows: Step 3.1: Calculate the inter-frame difference features. For the first... Frame skeleton features Calculate its skeleton features compared to the previous frame. Inter-frame difference features: (5); in, , Indicates the first Frame skeleton features Indicates the first Frame skeleton features Indicates the first Inter-frame difference features relative to the previous frame; when hour, Zero padding is used.

[0022] The inter-frame difference features on the complete time series are denoted as: (6); Step 3.2: Perform bi-branch projection and temporal pooling on the inter-frame difference features. Two independent branches are used. Convolutional branches and Perform a two-branch projection on the inter-frame difference features to obtain a first projection feature and a second projection feature: (7); (8); in, .

[0023] The first and second projected features are average pooled along the time dimension to obtain the time-pooled difference features: (9); (10); in, , This represents the number of feature channels after bi-branch projection, in this embodiment... .

[0024] Step 3.3: Construct the difference relation matrix. Based on the difference features after time pooling, perform matrix multiplication and Softmax normalization to obtain the difference relation matrix: (11); in, This represents the difference relation matrix generated from inter-frame difference features. This represents the number of feature channels after bi-branch projection, in this embodiment... .

[0025] Step 3.4: Merge the difference relation matrix into the first... Frame adjacency matrix. The difference relation matrix... Merging into the first Frame adjacency matrix The adjacency matrix after difference enhancement is obtained: (12); in, , This represents the fusion weights of the difference relation matrix. Because... Excluding batch size dimensions, while Generated from samples, actual fusion will... After broadcasting along the batch of large and small dimensions, and with Add them together.

[0026] Step 4: Based on the skeleton features after channel grouping, construct the attention relation matrix using Attention Relationship Units (ARUs). The calculation process of the attention relation unit (ARU) is as follows: Figure 4 As shown, channel grouping allows different feature subspaces to learn the joint association strength separately, avoiding the weakening of fine-grained joint co-expression by all channels sharing the same relational pattern.

[0027] The process of step four is as follows: Step 4.1: Group the input skeleton features by channel. The Attention Relationship Unit (ARU) groups the input skeleton features... Divided along the channel dimension into The channel group, the first The number of channels in each channel group is ,in, Indicates the channel group index. (Number) The feature representation corresponding to each channel group is as follows: (13); Step 4.2: Generate within-group query features and key features. For the first... Each channel group is averaged along the time dimension to obtain the intra-group time pooling features: (14); in, .

[0028] Based on the aforementioned intra-group time pooling characteristics, through intra-group... The convolutional branches generate query features respectively. Bond features : (15); (16); in, , Indicates the first Query characteristics of each channel group Indicates the first Key features of each channel group.

[0029] Step 4.3: Calculate the intra-group attention relationship matrix. Based on the... Calculate the scaled dot product attention relation matrix based on the query features and key features of each channel group: (17); in, Indicates the first The intra-group attention relationship matrix corresponding to each channel group.

[0030] Step 4.4: Average and fuse multiple attention relationship matrices. For all... The attention relation matrix of each channel group is averaged to obtain the attention relation matrix output by the attention relation unit ARU: (18); in, , represents the attention relation matrix output by the attention relation unit ARU.

[0031] Step 5: The first... Frame adjacency The difference relation matrix and the attention relationship matrix By merging, the final result is obtained. Frame-enhanced adjacency matrix: (19); in, , The fusion weights represent the difference relation matrix. Represents the fusion weights of the attention relationship matrix; This only represents the difference matrix of the DRU output. This only represents the attention relation matrix output by the ARU.

[0032] because Excluding batch size dimensions, while All samples are generated from the original samples; during actual fusion, the samples will be... After broadcasting along the batch size dimension, the two are added together. This fusion method adopts the residual enhancement idea, which supplements two types of relationship matrices on the basis of preserving the existing frame-level topological relationships, so that the final adjacency matrix simultaneously contains structural topology, motion change and channel attention information.

[0033] Step Six: The final... Frame Enhanced Adjacency Matrix The algorithm embeds TCN-GCN units and obtains gesture classification results based on graph convolutional feature extraction, global average pooling, fully connected layers, and Softmax. Therefore, the enhanced adjacency matrix does not directly replace the classification head, but plays a role in the graph convolutional feature aggregation stage, making the spatiotemporal skeleton features obtained by graph convolution aggregation more discriminative, and the subsequent pooling and classification layers complete the gesture category output.

[0034] The process of step six is ​​as follows: Step 6.1: Embed the final enhanced adjacency matrix into the TCN-GCN unit. The final... Frame Enhanced Adjacency Matrix The spatial graph convolutional path embedded in the TCN-GCN unit is used for relation aggregation and spatiotemporal feature extraction of skeleton keypoint features. The differential attention adjacency enhancement structure in the TCN-GCN unit is as follows: Figure 5 As shown.

[0035] Enhancing the data flow and tensor dimensions of the TD-GCN network, such as Figure 6 As shown.

[0036] Step 6.2: Classify and output the spatiotemporal features. After the graph convolutional features are aggregated by the final enhanced adjacency matrix, the skeleton spatiotemporal features are obtained. The skeleton spatiotemporal features are then processed sequentially through global average pooling, fully connected layers, and Softmax to obtain the dynamic gesture category probability distribution. The category with the highest probability is output as the gesture recognition result.

[0037] Specifically, the spatiotemporal features output by the last TCN-GCN unit are subjected to global average pooling along the time and keypoint dimensions to obtain pooled features. The pooled features are input into the fully connected layer to obtain the unnormalized classification score. (20); in, This represents the result after global average pooling. 3D feature vectors This represents the weight matrix of the fully connected layer. Indicates the bias term. This represents the unnormalized classification score output by the fully connected layer. Indicates the number of gesture categories.

[0038] Then obtain the first value through Softmax. Predicted probability of gestures: (twenty one); in, Indicates the gesture category index. Indicates the first The predicted probability of gestures, and The final predicted category is: (twenty two); in, This indicates the final predicted category.

[0039] It should be noted that the above formula describes the classification process of a single sample; when implemented with batch size, the batch features after global average pooling can be expressed as follows: The output of a fully connected layer can be represented as After Softmax, the batch class probability distribution is obtained. .

[0040] To verify the performance of the method of this invention, a comparative experiment was conducted on the SHREC'17 14-class dynamic gesture dataset. This dataset is a dynamic gesture recognition dataset, and the experimental input is a three-dimensional skeleton sequence composed of 22 hand joints, with an original coordinate channel count of [missing information]. The time frame number is uniformly sampled as The number of gesture categories is This embodiment sets up two experiments under the same skeleton gesture recognition task: Experiment 1 is used to compare the baseline model without enhancement modules, the differential relation enhancement model, and the differential relation and channel attention enhancement model; Experiment 2 is used to examine the differential relation fusion weights. Impact on bone modality recognition results.

[0041] The experimental environment for this embodiment is as follows: the operating system is Ubuntu 20.04.1 LTS, the central processing unit is an Intel Xeon Gold 6430 with 32 computing cores, and the graphics processor is an NVIDIA GeForce RTX 4090 with 24GB of video memory. The experimental program is written in Python 3.11.7, the deep learning framework is PyTorch 2.5.1+cu124, the runtime CUDA version is 12.4, the nvcc compiler version is 12.1, and the cuDNN version is 9.1.0. The main training parameters are shown in Table 1 below.

[0042] Table 1 shows the training parameters;

[0043] Experiment 1: Ablation Experiment of Differential Relational Unit and Attention Relational Unit. Experiment 1 is used to verify the impact of the Differential Relational Unit (DRU) and Attention Relational Unit (ARU) on the accuracy of skeleton gesture recognition in this invention. The experiment consists of three sets of models: the first set is the baseline model without any enhancement modules, which only uses the existing frame-level adjacency matrix for graph convolutional feature aggregation; the second set is the differential relational enhancement model, which uses the existing frame-level adjacency matrix for graph convolutional feature aggregation; Introducing a difference relation matrix based on the frame adjacency matrix The third group consists of a difference relation and channel attention enhancement model, which further introduces an attention relation matrix on top of the difference relation enhancement. And construct the final enhanced adjacency matrix. The structural configurations and recognition results of the three models are combined in Table 2.

[0044] Table 2 shows the results of the ablation experiment of the difference relation unit and the attention relation unit;

[0045] As shown in Table 2, in the existing... Adding DRU to the frame adjacency matrix improved the bone modality recognition accuracy from 87.62% to 89.88%, an improvement of +2.26%. This indicates that the difference relation matrix constructed based on inter-frame difference features can effectively supplement the missing motion change information in the existing frame-level adjacency matrix, especially beneficial for modeling bone structure changes, local motion directions, and relative joint displacement relationships. Further adding ARU improved the joint modality recognition accuracy from 96.79% to 97.26%, an improvement of +0.47%. This demonstrates that the channel grouping attention relation matrix can further mine joint association patterns in different channel subspaces within the keypoint coordinate features, thereby enhancing the fine-grained joint relationship expression capability in the joint modality. From the results of different modalities, DRU is more sensitive to bone modalities, while ARU is more effective for joint modalities. Therefore, in the preferred embodiment, when the recognition task uses joint input, the differential relation and channel attention enhancement model is preferred; when the recognition task uses bone input, the differential relation enhancement model is preferred. The above results demonstrate that DRU and ARU enhance the adjacency matrix representation capability from the perspectives of inter-frame motion differential relations and channel group attention relations, respectively, and the two have complementary effects on different input modalities.

[0046] Experiment 2: Fusion weight ablation experiment. Experiment 2 is used to verify the fusion weight of the difference relation matrix. Impact on bone modality recognition results. Figure 7 Given Results of the fusion weight ablation experiment.

[0047] Figure 7 It includes four ablation settings, denoted as ABL-A0, ABL-A1, ABL-A2, and ABL-A3. For ease of explanation, they can be described as the following four schemes. ABL-A0 represents the "zero-weight fixed-difference fusion scheme". This scheme will... Setting it to 0 and never updating it is equivalent to turning off the difference relation matrix. The contribution to the final adjacency matrix is ​​used to verify whether the difference relation term itself is necessary; ABL-A1 represents "recommended learnable difference fusion scheme". This scheme will... Initialized to 0.1, and updated as a learnable parameter during training. This setting enables the model to adaptively adjust the contribution of the inter-frame difference relation matrix to the final adjacency matrix during training, and is the difference fusion weight setting in the preferred embodiment of this invention; ABL-A2 represents the "fixed small weight difference fusion scheme". This scheme will... Initialized to 0.1, but this parameter is not updated during training. This setting is used to distinguish between "introducing non-zero difference terms" and "learnable adjustable difference contribution," verifying whether simply adding a fixed difference term is sufficient to bring stable improvements; ABL-A3 indicates a "learnable difference fusion scheme with large initial values." This scheme will... Initialize to 0.5 and update as a learnable parameter during training. This setting is used to verify whether excessively large initial differential weights will interfere with the utilization of existing frame-level topological relationships.

[0048] Experimental results show that ABL-A1 converges faster in the early stages of training and achieves the best performance in the final test, reaching 89.88%. The final bone modality test results for ABL-A0, ABL-A2, and ABL-A3 are all 87.26%, a decrease of 0.36 percentage points compared to the baseline model without enhancement modules. This phenomenon indicates that disabling the difference relation term, fixing the difference weights, or using excessively large learnable initial weights are all insufficient to stably improve bone modality recognition performance. Therefore, this invention preferably uses... Initialize it to 0.1 and set it as a learnable parameter.

[0049] The two experiments above demonstrate that the present invention, by introducing DRU on top of the existing frame-level adjacency matrix, can significantly improve the dynamic gesture recognition performance in the bone modality; further, by introducing ARU, it can improve the recognition performance in the joint modality. Experimental results prove that the skeleton gesture recognition method based on differential relationships and channel attention described in this invention can enhance the expressive power of the adjacency matrix from two aspects: inter-frame motion difference and channel group attention, and improve the skeleton gesture recognition effect with a small increase in the number of parameters. Specifically, DRU is more sensitive to the bone modality and is suitable for supplementing changes in skeletal structure and local motion relationships; ARU is more effective for the joint modality and is suitable for supplementing channel subspace joint relationships in the keypoint coordinate features. Therefore, this invention can select a more suitable single-stream model configuration according to different input modalities: the joint modality preferentially uses the differential relationship and channel attention enhancement model, while the bone modality preferentially uses the differential relationship enhancement model.

[0050] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A skeleton gesture recognition method based on differential relations and channel attention, characterized in that, The method includes the following steps: Step 1: Obtain the skeleton sequence of the dynamic gesture to be recognized, and represent the skeleton sequence as an input tensor; Step 2: Based on the skeleton sequence, obtain the first skeleton graph convolutional network. Frame adjacency matrix and the first Frame adjacency matrix As a common enhancement object for Differential Relational Unit (DRU) and Attention Relational Unit (ARU); Step 3: Based on the skeleton feature differences between adjacent time frames, construct a difference relation matrix using Differential Relationship Units (DRUs). ; Step 4: Based on the skeleton features after channel grouping, construct the attention relation matrix using Attention Relationship Units (ARUs). ; Step 5: The first... Frame adjacency matrix The difference relation matrix and the attention relationship matrix By merging, the final result is obtained. Frame Enhanced Adjacency Matrix ; Step Six: The final... Frame Enhanced Adjacency Matrix The TCN-GCN unit is embedded, and the gesture classification results are obtained based on graph convolution feature extraction, global average pooling, fully connected layers and Softmax.

2. The skeleton gesture recognition method based on differential relationship and channel attention as described in claim 1, characterized in that, In step one, the input tensor is: (1) in, , Indicates batch size, Indicates the number of feature channels. Indicates the number of time frames. This indicates the number of joints in the skeleton.

3. The skeleton gesture recognition method based on differential relationship and channel attention as described in claim 1 or 2, characterized in that, The process of step two is as follows: Step 2.1: Construct the basic adjacency matrix, the first... Frame adjacency matrix It can be obtained by combining a predefined skeleton adjacency matrix, a learnable prior adjacency matrix, channel topology residuals, and frame-level topology residuals. The predefined skeleton adjacency matrix... and learnable prior adjacency matrix Construct the basic adjacency matrix: （2）； in, ; Step 2.2: Introduce channel topology residuals into the basic adjacency matrix, and then... By fusing these matrices into the basic adjacency matrix, a channel decoupling adjacency matrix is ​​obtained: （3）； in, , , The fusion coefficient represents the channel topology residual; Step 2.3: Introduce the first [unclear] into the channel decoupling adjacency matrix. The frame-level topology residual corresponding to the frame, will be the frame-level topology residual. The result is obtained by fusing the decoupled adjacency matrix of the channel. Frame adjacency matrix: （4）； in, , The fusion coefficients represent the frame-level topology residuals. This invention constructs an adjacency matrix for the link output based on an existing frame-level adjacency matrix, and uses it as the basis for enhancing subsequent differential and attention relationships.

4. The skeleton gesture recognition method based on differential relationship and channel attention as described in claim 1 or 2, characterized in that, The process of step three is as follows: Step 3.1: Calculate the inter-frame difference features. For the first frame... Frame skeleton features Calculate its skeleton features compared to the previous frame. Inter-frame difference features: （5）； in, , Indicates the first Frame skeleton features Indicates the first Frame skeleton features Indicates the first Inter-frame difference features relative to the previous frame; when hour, Use zero padding; The inter-frame difference features on the complete time series are denoted as: （6）； Step 3.2: Perform bi-branch projection and temporal pooling on the inter-frame difference features, using two independent... Convolutional branches and Perform a two-branch projection on the inter-frame difference features to obtain a first projection feature and a second projection feature: （7）； （8）； in, ; The first and second projected features are average pooled along the time dimension to obtain the time-pooled difference features: （9）； （10）； in, , This represents the number of feature channels after bibranch projection; Step 3.3: Construct the difference relation matrix. Based on the difference features after time pooling, perform matrix multiplication and Softmax normalization to obtain the difference relation matrix: （11）； in, This represents the difference relation matrix generated from inter-frame difference features. This represents the number of feature channels after bibranch projection; Step 3.4: Merge the difference relation matrix into the first... The frame adjacency matrix, which is the difference relation matrix Merging into the first Frame adjacency matrix The adjacency matrix after difference enhancement is obtained: （12）； in, , This represents the fusion weights of the difference relation matrix.

5. The skeleton gesture recognition method based on differential relationship and channel attention as described in claim 1 or 2, characterized in that, The process of step four is as follows: Step 4.1: Perform channel grouping on the input skeleton features. The Attention Relationship Unit (ARU) will then group the input skeleton features... Divided along the channel dimension into The channel group, the first The number of channels in each channel group is ,in, Indicates the channel group index, number 1 The feature representation corresponding to each channel group is as follows: （13）； Step 4.2: Generate within-group query features and key features for the first group. Each channel group is averaged along the time dimension to obtain the intra-group time pooling features: （14）； in, ; Based on the aforementioned intra-group time pooling characteristics, through intra-group... The convolutional branches generate query features respectively. Bond features : （15）； （16）； in, , Indicates the first Query characteristics of each channel group Indicates the first Key features of each channel group; Step 4.3: Calculate the intra-group attention relationship matrix, based on the first... Calculate the scaled dot product attention relation matrix based on the query features and key features of each channel group: （17）； in, Indicates the first The intra-group attention relationship matrix corresponding to each channel group; Step 4.4: Average and fuse multiple attention relationship matrices, and then perform a full fusion. The attention relation matrix of each channel group is averaged to obtain the attention relation matrix output by the attention relation unit ARU: （18）； in, , represents the attention relation matrix output by the attention relation unit ARU.

6. The skeleton gesture recognition method based on differential relationship and channel attention as described in claim 1 or 2, characterized in that, In step five, the first Frame adjacency The difference relation matrix and the attention relationship matrix By merging, the final result is obtained. Frame-enhanced adjacency matrix: （19）； in, , The fusion weights represent the difference relation matrix. Represents the fusion weights of the attention relationship matrix; This only represents the difference matrix of the DRU output. This only represents the attention relation matrix output by the ARU.

7. The skeleton gesture recognition method based on differential relationship and channel attention as described in claim 1 or 2, characterized in that, The process of step six is ​​as follows: Step 6.1: Embed the final enhanced adjacency matrix into the TCN-GCN unit, and embed the final... Frame Enhanced Adjacency Matrix Spatial graph convolutional paths embedded in TCN-GCN units are used to perform relation aggregation and spatiotemporal feature extraction on skeleton keypoint features; Step 6.2: Classify and output the spatiotemporal features. After the graph convolutional features are aggregated by the final enhanced adjacency matrix, the skeleton spatiotemporal features are obtained. The skeleton spatiotemporal features are then processed by global average pooling, fully connected layers and Softmax to obtain the dynamic gesture category probability distribution. The category with the highest probability is output as the gesture recognition result.

8. The skeleton gesture recognition method based on differential relationship and channel attention as described in claim 1 or 2, characterized in that, In step 6.2, the spatiotemporal features output by the last TCN-GCN unit are subjected to global average pooling along the time and keypoint dimensions to obtain pooled features. The pooled features are input into the fully connected layer to obtain the unnormalized classification score. （20）； in, This represents the result after global average pooling. 3D feature vectors This represents the weight matrix of the fully connected layer. Indicates the bias term. This represents the unnormalized classification score output by the fully connected layer. M Indicates the number of gesture categories; Then obtain the first value through Softmax. Predicted probability of gestures: （21）； in, Indicates the gesture category index. Indicates the first The predicted probability of gestures, and The final predicted category is: （22）； in, This indicates the final predicted category.

Images