Dynamic prompt decoupling Transform-based skeleton human body action fine-grained recognition method

Abstract

The invention relates to a skeleton human body action fine-grained recognition method based on dynamic prompt decoupling Transform. Comprising the steps of (1) data acquisition and preprocessing, (2) decoupling type visual Transform modeling, (3) dynamic text prompt generation, (4) visual-text semantic alignment, performing text-guided visual feature enhancement through a semantic adjustment module so as to bridge a modal gap between visual and text, and (5) multi-modal cooperative training. The visual enhancement features and the dynamic text features are aligned through comparative learning, semantic stability of the dynamic text features is kept through consistency learning, and action classification is completed through cross entropy loss. Through interaction of a cross-modal attention mechanism, text-guided visual feature enhancement is realized, so that a modal gap is effectively relieved, and priori knowledge of a text is converted into discrimination improvement of visual features.

Description

Technical Field

[0001] This invention belongs to the field of human motion recognition, and involves skeletal motion recognition, fine-grained behavior analysis, and multimodal learning. In particular, it relates to a fine-grained skeletal human motion recognition method based on dynamic prompts and decoupled Transformer. Background Technology

[0002] Human motion recognition is an important research topic with broad social impact. In the field of smart healthcare, systems track patients' movements in real time, quantitatively assess the completion of rehabilitation training, and then develop targeted rehabilitation plans. In the field of intelligent security, systems detect human movements to promptly identify emergencies such as intrusions and falls, helping security personnel react quickly. In the field of human-computer interaction, virtual reality achieves immersive experiences through motion mapping, and service robots improve service acceptance through natural motion interaction. The development of human motion recognition technology in these applications has greatly promoted the improvement of social life quality, thus possessing immense research value. Compared to image data, skeletal data is computationally efficient, highly resistant to interference, offers high privacy protection, and boasts excellent real-time performance; therefore, skeletal-based human motion recognition is gradually becoming a research hotspot.

[0003] Early methods based on convolutional neural networks (CNN) and recurrent neural networks (RNN) simply treated skeletal data as pseudo-images or time series. Although computationally efficient, they completely ignored the inherent topological connections of the human body, resulting in a lack of kinematic constraints. In recent years, graph convolutional networks (GCN) have made breakthroughs by explicitly modeling the graph structure (node-edge relationships) of human joints and bones. However, they rely on predefined adjacency matrices and are limited by the local receptive field of graph convolution, making it difficult to effectively capture the collaborative motion between remote joints (such as the "left hand-right foot" interaction). Currently, Transformer has become a new research direction due to its self-attention mechanism, but it still has three key problems in skeletal motion recognition: (1) First, the spatiotemporal dimension and the human interaction dimension are coupled and interfere with each other in the self-attention calculation, or are even ignored, resulting in insufficient fine-grained modeling in specific dimensions. (2) Second, traditional attention performs fully connected calculations on all input positions, causing local micro-motion features to be submerged in global calculations, lacking explicit modeling. (3) Finally, the skeletal data itself only contains joint coordinates and lacks contextual interaction semantics, making it easy to confuse similar actions that depend on object interaction, such as "shaking hands" and "handing over objects". Summary of the Invention

[0004] The purpose of this invention is to provide a fine-grained method for skeletal human motion recognition based on dynamic prompts and decoupled Transformer.

[0005] Dimensional decoupling: For problem 1, the Transformer architecture can be decoupled into three dimensions: space, time, and human interaction. The traditional self-attention mechanism can also be reconstructed in a more intuitive and statistically grounded way, thereby enabling explicit fine-grained modeling of actions within and between individuals.

[0006] Local Enhancement: For Problem 2, the Spatial Transformer borrows from graph convolution, enhancing local spatial awareness by fusing a fixed human body structure graph and a data-driven learnable graph. The Temporal Transformer enhances local temporal awareness by constructing multi-scale dilated temporal convolutional modules and fusing local temporal constraints.

[0007] Semantic Injection: For Problem 3, we design visually guided dynamic cues to capture subtle differences in actions and guide the generation of more discriminative text. Simultaneously, we introduce fixed templates to stabilize semantics. Purely dynamic generation may cause text descriptions to deviate from their semantic meaning due to visual noise; fixed templates provide a semantic benchmark, ensuring that the core semantics of the category are not lost. The resulting dynamic text features adapt to specific samples, capturing details and achieving fine-grained differentiation, while maintaining the essence of the category, providing stable anchors, and ensuring training robustness.

[0008] Cross-modal interaction between visual and textual elements: After semantic injection, a modal gap exists between text and visual elements due to their different information granularities. Therefore, a semantic adjustment module is designed to interact through a cross-modal attention mechanism to achieve text-guided visual feature enhancement. Through this mechanism, the semantic adjustment module enables visual features to query and selectively absorb high-level semantic information related to the text modality, thereby semantically enhancing the original skeletal sequence features. This process effectively alleviates the modal gap and transforms the prior knowledge of the text into an improvement in the discriminative power of visual features.

[0009] To achieve the above objectives, this invention provides a fine-grained method for skeletal human motion recognition based on dynamic prompting and decoupling from Transformer, comprising the following steps:

[0010] S1: Obtain the human 3D skeleton joint position dataset for training, perform noise filtering and standardization on the raw skeleton data, and construct the joint feature matrix. .

[0011] S2: Decoupled visual Transformer modeling. Constructing a spatial Transformer to extract spatial correlation features from the joint feature matrix. Design a spatial statistical attention reconstruction module. and local space constraints This enables fine-grained spatial modeling.

[0012] S3: Constructing a temporal Transformer to extract temporal evolution features of the joint feature matrix Design a multi-scale dilated temporal convolution module. Time-based attention reconstruction module and local time constraints This enables fine-grained time modeling.

[0013] S4: Construct a human interaction Transformer to extract behavioral interaction relationship features. Design a human interaction statistical attention reconstruction module. This enables fine-grained human interaction modeling.

[0014] S5: Fuses S2, S3, and S4 in a concatenated manner, while introducing residual connections to form a single Transformer unit. The model uses a total of one graph convolutional mapping layer and nine Transformer units to generate the final visual features. .

[0015] S6: Construct action-specific dynamic text prompts Based on visual features To guide the generation of dynamic text features using the pre-trained large language model CLIP. It provides sample-level fine-grained descriptions to compensate for the lack of fine-grained contextual information in skeletal data. Simultaneously, it constructs fixed template hints. Mean text features Provides category-level general descriptions to ensure prompts during learning. Without deviating from the semantics of the action itself,

[0016] S7: Visual feature enhancement for text guidance via semantic adjustment module Bridging the modal gap between visual and textual.

[0017] S8: Multimodal collaborative training using a contrastive loss function. Alignment visual enhancement features and dynamic text features Utilizing consistency loss constraint semantics not deviated Using cross-entropy loss Complete the action classification.

[0018] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0019] (1) A dimensionally decoupled Transformer architecture is proposed, which proposes a three-level decoupling of space-time-human interaction, independently modeling spatial key relationships, temporal dynamic evolution and multi-person interaction features, thus solving the problem of insufficient fine-grained modeling caused by dimensional coupling in traditional Transformers.

[0020] (2) Statistically driven attention reconstruction: Introducing statistical measures such as mean (smoothing trend), standard deviation (dynamic fluctuation), and maximum value (significant feature) to reconstruct self-attention, enhancing the model's interpretability for fine-grained action components.

[0021] (3) Local-global collaborative modeling: Local constraints are introduced into the spatial and temporal Transformers to enhance local perception and alleviate the problem of weak local modeling capabilities of traditional Transformers.

[0022] (4) Semantic enhancement of dynamic prompts: Design visually guided dynamic prompts to generate discriminative dynamic text features. Introduce fixed templates to stabilize the core semantics of dynamic text features. The resulting dynamic text features not only adapt to specific samples, capture details, and achieve fine-grained differentiation, but also maintain the essence of the category, provide stable anchor points, and ensure training robustness.

[0023] (5) Visual and text cross-modal interaction: Design a semantic adjustment module and interact through a cross-modal attention mechanism to achieve text-guided visual feature enhancement, thereby effectively alleviating the modal gap and transforming the prior knowledge of the text into the discriminative power of visual features. Attached Figure Description

[0024] Figure 1 The flowchart provided for this invention

[0025] Figure 2 The joint structure diagram provided by this invention

[0026] Figure 3 This is a spatial Transformer structure diagram provided by the present invention.

[0027] Figure 4 The temporal Transformer structure diagram provided by this invention

[0028] Figure 5 This is a structural diagram of the semantic adjustment module provided by the present invention.

[0029] Figure 6 The overall task flowchart provided by this invention

[0030] Figure 7 This invention provides a visualization result of the spatial importance of the Transformer.

[0031] Figure 8 This invention provides a visualization of the temporal importance of the Time Transformer.

[0032] Figure 9 The image shows the accuracy visualization results of the human interaction Transformer provided by this invention. Detailed Implementation

[0033] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0034] Example: See Figures 1-9 A fine-grained method for skeletal human motion recognition based on dynamic prompts and decoupling from Transformer, characterized by the following steps:

[0035] S1: Obtain the human 3D skeleton joint position dataset for training, perform noise filtering and standardization on the raw skeleton data, and construct the joint feature matrix. ,

[0036] S2: Decoupled visual Transformer modeling, constructing a spatial Transformer to extract spatial correlation features from the joint feature matrix in S1. Design a spatial statistical attention reconstruction module and local space constraints To achieve fine-grained spatial modeling

[0037] S3: Constructing a temporal Transformer to extract temporal evolution features of the joint feature matrix Design a multi-scale dilated temporal convolution module Time-based attention reconstruction module and local time constraints To achieve fine-grained time modeling

[0038] S4: Construct a human interaction Transformer to extract behavioral interaction relationship features. Design a human interaction statistical attention reconstruction module To achieve fine-grained human interaction modeling,

[0039] S5: S2, S3, and S4 are fused in a concatenated manner, and residual connections are introduced to form a single Transformer unit. The model uses a total of one graph convolutional mapping layer and nine Transformer units to generate the final visual features. ,

[0040] S6: Construct action-specific dynamic text prompts Based on visual features To guide the generation of dynamic text features using the pre-trained large language model CLIP. It provides sample-level fine-grained descriptions to compensate for the lack of fine-grained contextual information in skeletal data. Simultaneously, it constructs fixed template hints. Mean text features Provides category-level general descriptions to ensure prompts during learning. Without deviating from the semantics of the action itself,

[0041] S7: Visual feature enhancement for text guidance via semantic adjustment module Bridging the modal gap between visual and textual.

[0042] S8: Multimodal collaborative training using a contrastive loss function. Alignment visual enhancement features and dynamic text features Utilizing consistency loss constraint semantics not deviated Using cross-entropy loss Complete the action classification.

[0043] In step S1, two mainstream datasets are selected: NTU-RGB+D 60 and NTU-RGB+D 120, as training datasets for human 3D skeletal joint positions. The NTU-RGB+D dataset is a large-scale 3D human behavior dataset containing 56,880 skeletal motion video clips covering 60 action categories, collected from 40 different subjects using 3 different camera perspectives. NTU-RGB+D 120 is an extended version of the NTU-RGB+D dataset, containing 113,945 samples collected from 106 subjects using 32 camera configurations, covering 120 action categories. Each video clip contains up to 25 body joint 3D coordinate data from 2 subjects. The joint structure is as follows: Figure 2As shown, the joints are defined as follows: 1-base of spine, 2-middle of spine, 3-neck, 4-head, 5-left shoulder, 6-left elbow, 7-left wrist, 8-left hand, 9-right shoulder, 10-right elbow, 11-right wrist, 12-right hand, 13-left hip, 14-left knee, 15-left ankle, 16-left foot, 17-right hip, 18-right knee, 19-right ankle, 20-right foot, 21-shoulder spine, 22-left fingertip, 23-left thumb, 24-right fingertip, 25-right thumb. For the two datasets above, the following three-level noise filtering is performed: Subjects with 11 or fewer consecutive frames are removed based on frame length. Based on spatial distribution, subjects with abnormal X / Y diffusion ratios at joints are removed. Based on motion magnitude, subjects with the largest and second largest non-overlapping motion amplitudes are retained. The following standardization processing is performed on the above two datasets: Sequence-level normalization is performed with joint 2 of the first valid frame of each sequence as the origin to eliminate absolute position interference; joint coordinates are dynamically scaled by the distance between joints 1 and 21 to eliminate individual size differences; and all sequences are padded / truncated to a uniform length of 64 frames to meet the requirements of deep learning models for fixed input dimensions.

[0044] In step S2, the spatial Transformer is a component of the decoupled visual Transformer, which consists of a spatial statistical attention reconstruction module. and local space constraints Composition, enabling fine-grained spatial modeling, such as Figure 3 As shown. The spatial fine-grained composition of motion may contain multiple interpretable dimensions. Inspired by statistics, we found that certain statistical operations naturally fit these semantic features: the mean encodes smooth regions, suitable for representing relatively stable pose components in motion. When the joint coordinates change gradually, the mean operation can effectively extract the central trend of its spatial distribution. The standard deviation (Std) represents the complexity of feature distribution. A high standard deviation value indicates more significant pixel changes between channels, which directly corresponds to an increase in the complexity of motion patterns. The maximum value (Max) captures significant features. The maximum difference in spatial coordinates can locate the most active joints. Specifically, for the spatial statistical attention reconstruction module, assuming the joint feature matrix in step S1... After convolution mapping, the dimension is... Decompose it into average values ​​according to the channel dimension. Three sets of features were used, and the following statistical operations were applied: , dimension Then, the following spatial attention interaction is adopted: , dimension By combining different statistical measures, the model can more accurately capture the differentiated interaction patterns between multidimensional features. The modulation parameters mentioned above are used to generate the weights of the spatial statistical attention reconstruction module. ,in These are 2-dimensional learnable weight parameters. It is the softmax activation function. Compared with the traditional self-attention mechanism, spatial statistical attention reconstruction has the following advantages: (1) more flexible fine-grained modeling, which fully considers the fine-grained composition characteristics of the action in a specific dimension. (2) lower computational complexity, with the same input joint feature matrix. , Far below traditional attention (3) Better interpretability: Channel separation based on statistical information ensures clear semantic correspondence. To address the long-distance dependency modeling characteristics of the attention mechanism in the spatial Transformer, the following approach is adopted to enhance local spatial perception: local spatial constraints are integrated into the spatial statistical attention reconstruction module. ,in These represent a fixed human anatomy diagram and a data-driven learnable graph, respectively, used to establish relationships and the strength of those relationships between local neighbor nodes. The spatial association features output by the final spatial Transformer are described as follows: .

[0045] In step S3, the temporal Transformer is a component of the decoupled visual Transformer, which consists of a multi-scale dilated temporal convolution module. Time-based attention reconstruction module and local time constraints Composition to achieve fine-grained time modeling, such as Figure 4 As shown. Temporal statistical attention reconstruction follows the construction rules of spatial statistical attention reconstruction, but has different statistical meanings: the mean captures the average dynamic characteristics of the action, such as the overall rhythm difference between "walking" and "running". The maximum value reflects the peak dynamics of the action, such as the moment of takeoff in the action of "jumping". The standard deviation quantifies the intensity of dynamic fluctuations and can distinguish action patterns such as "uniform speed running" and "variable speed running". The weights of the temporal statistical attention reconstruction module are shown. Calculation process and Similar, but focusing on time-based attentional interactions, with the following dimensions. Similarly, to address the long-distance dependency modeling characteristics of the attention mechanism in the temporal Transformer, the following two complementary schemes are adopted to enhance local temporal awareness: (1) a multi-scale dilated temporal convolution module, using a three-level dilation rate ( Parallel convolutional branches capture local short-term dependencies: Then, weighted fusion will be performed: (2) The temporal statistical attention reconstruction module incorporates local temporal constraints. Based on the core assumption that the inherent correlation strength between adjacent frames is significantly higher than that between distant frames, the module is designed... :

[0046]

[0047]

[0048]

[0049] in, Represents the distance between two frames. Represents the indicator function, when the distance Time equals 1, Set to 3, Set as , It is a constant. The values ​​increase with increasing proximity to the main diagonal, thus forming a weight distribution with gradually decreasing intensity centered on the identity matrix. The final time evolution characteristics of the transformer output are described as follows: .

[0050] In step S4, the human interaction Transformer is a component of the decoupled visual Transformer, which is composed of the human interaction statistical attention reconstruction module. This approach, which aims to achieve fine-grained human interaction modeling, addresses the common oversight of human interaction dimensions. This often leads to the loss of crucial collaborative dynamic information in multi-person scenarios, hindering fine-grained behavior analysis. In the human interaction statistical attention reconstruction module, the mean is used to capture the overall collaborative trend among multiple actors, while the maximum value is used to identify the dominant individual. Unlike spatial and temporal transformers, this approach does not use standard deviation for statistical analysis because most interactive actions exhibit strong symmetry—for example, the behavioral differences between two individuals in a "handshake" action are minimal, causing the standard deviation to approach zero. Therefore, by integrating only the interaction relationship between the mean and the maximum value, the attention mechanism can be represented as follows: , dimension The final human interaction features output by the Human Interaction Transformer are described as follows: .

[0051] In step S5, the spatial Transformer, temporal Transformer, and human interaction Transformer are connected in series, and residual connections are introduced to form a single Transformer unit. The skeletal model contains a total of one graph convolutional mapping layer and nine Transformer units, with feature dimensions of 60-60-60-120-120-120-240-240-240, generating the final visual features. .

[0052] In step S6, given category information Construct action-specific dynamic text prompts ,in Represents a learnable cue vector. Indicates visual guidance, i.e., visual features After linear layer mapping to and Same dimension ( Set to 4) to inject sample-specific visual context. This dynamic cue, incorporating visual information, is fed into the pre-trained large language model CLIP to generate dynamic text features. This provides fine-grained semantic descriptions that adapt to samples, effectively compensating for the lack of contextual information such as objects and scenes in skeletal data. Simultaneously, to provide stable and robust category-level semantic anchors and prevent the model from deviating from core concepts due to overfitting dynamic details, category information is also generated using a set of predefined fixed template generators (including: "a photo of an action [CLASS]", "an image of an action [CLASS]", "a video of [CLASS]", "video classification [CLASS]", "[CLASS], an action", "[CLASS], this is an action", "human action [CLASS]", "the action of playing [CLASS]", "[CLASS]") to generate various fixed templates. The features obtained from these templates using CLIP are averaged to generate the mean text feature. The fixed template feature, as a semantic regularization term, works in conjunction with the dynamic text features during training. The dynamic features enable the model to capture subtle differences in actions, while the fixed features ensure that the model firmly grasps the overall definition and core semantics of the category. The combination of the two achieves a balance between fine-grained discrimination and category stability.

[0053] In step S7, a semantic adjustment module is designed to bridge the modal gap between visual and textual features, enhancing visual features using textual features, such as... Figure 5 As shown, due to the fundamental differences in the original data structure and semantic abstraction level between text features and visual features, directly forcing alignment will lead to the loss of modality-specific information. Therefore, given text features... and visual features A cross-modal attention mechanism is used to achieve modal interaction, specifically as follows: features are divided into... Groups, apply a linear transformation to each group of features:

[0054]

[0055]

[0056]

[0057] in, Represents a linear transformation. Set to 8, then the cross-modal interaction is expressed as:

[0058]

[0059]

[0060] This design enables visual features to actively "query" relevant textual semantics to enhance their representation, while preserving the original skeletal features. Enhanced features generated by concatenating cross-group features to prevent information loss After transformation through two more global projection layers, the dimensions are unified with the text features, thus supporting effective alignment for downstream contrastive training.

[0061] .

[0062] In step S8, visual enhancement features and text features Projected onto a unified feature space, and optimized using a bidirectional contrastive learning loss, such as... Figure 6 As shown:

[0063]

[0064] in:

[0065]

[0066]

[0067] in Represents the similarity function. This is the temperature parameter; set it to 0.8.

[0068] Preserving dynamic text features Semantic stability, employing consistency loss constraints and mean text features :

[0069]

[0070] At the same time, cross-entropy loss is introduced to guide action classification:

[0071]

[0072] in, Based on The predicted action category. By combining the two losses, the final multi-task loss used for this task is:

[0073]

[0074] in, This represents the weighting parameter, all of which are set to 0.5.

[0075] To demonstrate the effectiveness of this invention, experimental results are provided on the NTU-RGB+D 60 and NTU-RGB+D 120 datasets. The experimental results are shown in Table 1.

[0076] Table 1 Comparison of results from different methods on NTU-RGB+D 60 and NTU-RGB+D 120 datasets.

[0077]

[0078] The table primarily reports the Top-1 accuracy under different individual (Xsub) and different viewpoint (Xview) evaluation modes, with bold indicating the highest accuracy. As shown in the table, the AGC-LSTM method in row 1 uses an RNN approach, resulting in an excessively large parameter count of 22.9M. Rows 2-8 employ the GCN method, benefiting from the local modeling advantages of graph convolution, thus improving accuracy with a reduced parameter count. Rows 9-12 use the Transformer method, benefiting from the global modeling advantages of the Transformer, also resulting in improved accuracy. Finally, this patented method, by decoupling the Transformer, remains competitive with mainstream methods despite having only 2.0M parameters.

[0079] To more intuitively verify the effectiveness of this method, a visual analysis was performed on the spatial, temporal, and human interaction Transformers. Figure 7 The diagram illustrates the spatial importance that the Spatial Transformer focuses on, with larger red circles indicating higher levels of attention. The diagram reveals that the Spatial Transformer considers both simple joint movements (such as "salute") and complex full-body movements (such as "taking off a shoe"). Figure 8 This demonstrates the temporal importance that the Temporal Transformer focuses on. Two sets of temporally inverse actions are selected for visualization analysis. If the model lacks a robust understanding of fine-grained temporal features, it is highly susceptible to misclassification. Taking "take jacket" and "take off jacket" as examples, this model assigns higher temporal attention weights to keyframes in the dressing and undressing processes. This phenomenon indicates that the model can effectively capture the fine-grained temporal dynamics of such actions, especially focusing on the transition phases where the action state undergoes significant changes. Figure 9 The effectiveness of the human interaction Transformer was visualized. Here, categories with high levels of multi-person interaction were specifically chosen for accuracy visualization. "Blue" indicates that the human interaction Transformer is not involved, while "orange" indicates it is involved. It can be observed that orange indicates a significant increase in accuracy.

[0080] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A fine-grained method for skeletal human motion recognition based on dynamic prompts and decoupled Transformer, characterized in that, The method includes the following steps: S1: Obtain the human 3D skeleton joint position dataset for training, perform noise filtering and standardization on the raw skeleton data, and construct the joint feature matrix. , S2: Decoupled visual Transformer modeling, constructing a spatial Transformer to extract spatial correlation features from the joint feature matrix in S1. Design a spatial statistical attention reconstruction module and local space constraints To achieve fine-grained spatial modeling S3: Constructing a temporal Transformer to extract temporal evolution features of the joint feature matrix Design a multi-scale dilated temporal convolution module Time-based attention reconstruction module and local time constraints To achieve fine-grained time modeling S4: Construct a human interaction Transformer to extract behavioral interaction relationship features. Design a human interaction statistical attention reconstruction module To achieve fine-grained human interaction modeling, S5: S2, S3, and S4 are fused in a concatenated manner, and residual connections are introduced to form a single Transformer unit. The model uses a total of one graph convolutional mapping layer and nine Transformer units to generate the final visual features. , S6: Construct action-specific dynamic text prompts Based on visual features To guide the generation of dynamic text features using the pre-trained large language model CLIP. It provides sample-level fine-grained descriptions to compensate for the lack of fine-grained contextual information in skeletal data, and at the same time, constructs fixed template hints. Mean text features Provides category-level general descriptions to ensure prompts during learning. Without deviating from the semantics of the action itself, S7: Visual feature enhancement for text guidance via semantic adjustment module Bridging the modal gap between visual and textual. S8: Multimodal collaborative training using a contrastive loss function. Alignment visual enhancement features and dynamic text features Utilizing consistency loss constraint semantics not deviated Using cross-entropy loss Complete the action classification.

2. The fine-grained skeletal human motion recognition method based on dynamic prompting and decoupling from Transformer as described in claim 1, characterized in that: In step S1, two mainstream datasets are selected: NTU-RGB+D 60 and NTU-RGB+D 120, as training datasets for human 3D skeletal joint positions. The NTU-RGB+D dataset is a large-scale 3D human behavior dataset containing 56,880 skeletal motion video clips covering 60 action categories, collected from 40 different subjects using 3 different camera perspectives. NTU-RGB+D 120 is an extended version of the NTU-RGB+D dataset, containing 113,945 samples from 106 subjects using 32 camera configurations, covering 120 action categories. Each video clip contains 3D coordinate data for up to 25 body joints from 2 subjects. Each joint is defined as: 1-base of spine, 2-middle of spine, 3-neck, 4-head, 5-left shoulder, ... For the two datasets above, the following three-level noise filtering is performed: 6-left elbow, 7-left wrist, 8-left hand, 9-right shoulder, 10-right elbow, 11-right wrist, 12-right hand, 13-left hip, 14-left knee, 15-left ankle, 16-left foot, 17-right hip, 18-right knee, 19-right ankle, 20-right foot, 21-shoulder spine, 22-left fingertip, 23-left thumb, 24-right fingertip, 25-right thumb. Based on spatial distribution, subjects with abnormal X / Y diffusion ratios at joints are removed. Based on motion magnitude, subjects with the largest and second largest non-overlapping motion amplitudes are retained. The following standardization processing is performed on the above two datasets: Sequence-level normalization is performed with joint 2 of the first valid frame of each sequence as the origin to eliminate absolute position interference; joint coordinates are dynamically scaled by the distance between joints 1 and 21 to eliminate individual size differences; and all sequences are padded / truncated to a uniform length of 64 frames to meet the requirements of deep learning models for fixed input dimensions.

3. The fine-grained skeletal human motion recognition method based on dynamic prompting and decoupling from Transformer as described in claim 1, characterized in that: In step S2, the spatial Transformer is a component of the decoupled visual Transformer, which consists of a spatial statistical attention reconstruction module. and local space constraints This structure enables fine-grained spatial modeling. When the joint coordinates change gradually, the mean operation can effectively extract the central trend of its spatial distribution. The standard deviation (Std) characterizes the complexity of the feature distribution; a high standard deviation indicates more significant pixel changes between channels, which directly corresponds to an increase in the complexity of the motion pattern. The maximum value (Max) captures significant features; the difference in the maximum value of spatial coordinates can locate the most active joints. Specifically, for the spatial statistical attention reconstruction module, assuming the joint feature matrix in step S1... After convolution mapping, the dimension is... Decompose it into average values ​​according to the channel dimension. Three sets of features were used, and the following statistical operations were applied: , dimension Then, the following spatial attention interaction is adopted: , dimension By combining different statistical measures, the model can more accurately capture the differentiated interaction patterns between multidimensional features. The modulation parameters mentioned above are used to generate the weights of the spatial statistical attention reconstruction module. ,in These are 2-dimensional learnable weight parameters. It uses the softmax activation function. To address the long-distance dependency modeling characteristics of the attention mechanism in spatial Transformers, the following approach is adopted to enhance local spatial awareness: local spatial constraints are incorporated into the spatial statistical attention reconstruction module. ,in These represent a fixed human anatomy diagram and a data-driven learnable graph, respectively, used to establish relationships and the strength of those relationships between local neighbor nodes. The spatial association features output by the final spatial Transformer are described as follows: .

4. The fine-grained skeletal human motion recognition method based on dynamic prompting and decoupling from Transformer as described in claim 1, characterized in that: In step S3, the temporal Transformer is a component of the decoupled visual Transformer, which consists of multi-scale dilated temporal convolution modules. Time-based attention reconstruction module and local time constraints Composition to achieve fine-grained temporal modeling. In view of the long-distance dependency modeling characteristics of the attention mechanism in the temporal Transformer, the following two complementary schemes are adopted to enhance local temporal awareness: (1) Multi-scale dilated temporal convolution module, using a three-level dilation rate ( Parallel convolutional branches capture local short-term dependencies: Then, weighted fusion will be performed: (2) The temporal statistical attention reconstruction module incorporates local temporal constraints. Based on the core assumption that the inherent correlation strength between adjacent frames is significantly higher than that between distant frames, the module is designed... : in, Represents the distance between two frames. Represents the indicator function, when the distance Time equals 1, Set to 3, Set as , It is a constant. The values ​​increase with increasing proximity to the main diagonal, thus forming a weight distribution with gradually decreasing intensity centered on the identity matrix. The final time evolution characteristics of the transformer output are described as follows: .

5. The fine-grained skeletal human motion recognition method based on dynamic prompting and decoupled Transformer according to claim 1, characterized in that: In step S4, the human interaction Transformer is a component of the decoupled visual Transformer, which is composed of the human interaction statistical attention reconstruction module. This approach constructs a fine-grained model of human interaction. In the human interaction statistical attention reconstruction module, it uses the mean to capture the overall collaborative trend among multiple actors and uses the maximum value to identify the dominant individual. Unlike spatial and temporal transformers, it does not use standard deviation for statistics; it only integrates the interaction relationship between the mean and the maximum value. Its attention mechanism can be represented as follows: , dimension The final human interaction features output by the Human Interaction Transformer are described as follows: .

6. The fine-grained skeletal human motion recognition method based on dynamic prompting and decoupling of Transformer according to claim 1, characterized in that: In step S5, the spatial Transformer, temporal Transformer, and human interaction Transformer are connected in series, and residual connections are introduced to form a single Transformer unit. The skeletal model contains a total of one graph convolutional mapping layer and nine Transformer units, with feature dimensions of 60-60-60-120-120-120-240-240-240, generating the final visual features. .

7. The fine-grained skeletal human motion recognition method based on dynamic prompting and decoupling from Transformer according to claim 1, characterized in that: In step S6, given category information Construct action-specific dynamic text prompts ,in Represents a learnable cue vector. Indicates visual guidance, i.e., visual features After linear layer mapping to and Same dimension ( Set to 4) to inject sample-specific visual context, while simultaneously constructing fixed template cues. Mean text features Provides category-level general descriptions to ensure prompts during learning. Without deviating from the semantics of the action itself.

8. The fine-grained skeletal human motion recognition method based on dynamic prompting and decoupled Transformer according to claim 1, characterized in that: In step S7, a semantic adjustment module is designed to bridge the modal gap between visual and textual features. This module enhances visual features using textual features. However, due to the fundamental differences in their original data structures and semantic abstraction levels, directly forcing alignment between textual and visual features would lead to the loss of modality-specific information. Therefore, given textual features… and visual features A cross-modal attention mechanism is used to achieve modal interaction, specifically as follows: features are divided into... Groups, apply a linear transformation to each group of features: in, Represents a linear transformation. Set to 8, then the cross-modal interaction is expressed as: This design enables visual features to actively "query" relevant textual semantics to enhance their representation, while preserving the original skeletal features. Enhanced features generated by concatenating cross-group features to prevent information loss After transformation through two more global projection layers, the dimensions are unified with the text features, thus supporting effective alignment for downstream contrastive training. 。 9. The fine-grained skeletal human motion recognition method based on dynamic prompting and decoupling from Transformer according to claim 1, characterized in that: In step S8, visual enhancement features and text features Projected onto a unified feature space, and optimized using bidirectional contrastive learning loss. in: in Represents the similarity function. This is the temperature parameter; set it to 0.

8. Preserving dynamic text features Semantic stability, employing consistency loss constraints and mean text features : At the same time, cross-entropy loss is introduced to guide action classification: in, Based on The predicted action category, combined with the losses from all three factors, results in the following multi-task loss for this task: in, This represents the weighting parameter, all of which are set to 0.

5.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the fine-grained skeletal human motion recognition method based on dynamic prompting and decoupling of Transformer as described in any one of claims 1 to 9.

Images