A self-supervised skeleton behavior recognition method, system, device, and medium based on action semantic guidance

Abstract

A self-supervised skeleton behavior recognition method, system, device, and medium based on action semantic guidance are disclosed. The method involves: first, acquiring and preprocessing skeleton video samples to be trained; second, generating semantic prototypes for each type of action in the preprocessed human skeleton dataset using an action semantic generation module, i.e., generating corresponding action text semantic features; third, inputting the preprocessed human skeleton dataset into a feature enhancement module (FAM) to enhance the skeleton data and generate corresponding action skeleton features; and finally, fusing the action text semantic features and skeleton features using a multimodal fusion module to assist the model in learning more distinctive and representative action representations from a semantic level, improving the accuracy of fuzzy action recognition. Through contrastive learning, a self-supervised approach aims to ensure model recognition performance while reducing dependence on large-scale labeled data, achieving skeleton-based human action classification. The system, device, and medium, based on the action semantic guidance self-supervised skeleton behavior recognition method, identify and classify human actions with high accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of skeleton-based human motion recognition technology, and in particular to a self-supervised skeleton behavior recognition method, system, device and medium based on action semantic guidance. Background Technology

[0002] Actions and behaviors fall under the category of human biometrics, and recognizing and understanding the actions and behaviors of observed individuals is a fundamental psychological attribute of human visual perception and cognition. Human motion recognition, as a basic and important topic in computer vision, has a wide range of applications, including video retrieval, human-computer interaction, sign language recognition, and autonomous driving. Furthermore, behavior recognition plays a crucial role in individual identification, communication interaction, and inferring the behavioral intentions of others.

[0003] With the rapid development of depth sensors and pose estimation algorithms, 3D skeleton data has become easier to acquire. Furthermore, unlike data from other modalities, compact skeleton data provides specific location and motion information of human joints, facilitating spatiotemporal and motion reconstruction, avoiding interference from the background environment, and effectively improving the accuracy and robustness of action recognition in noisy scenes. In the past few years, self-supervised contrastive learning skeleton behavior recognition methods have achieved excellent performance; however, these methods still face many challenges: First, existing skeleton data augmentation strategies cannot provide rich, difficult contrast pairs for contrastive learning to effectively learn general discriminative spatiotemporal skeleton motion representations; second, skeleton data lacks contextual information and interactive objects, and the differences between similar action classes are small. Therefore, directly modeling based on skeleton spatiotemporal features offers limited performance gains for indistinguishable fuzzy behaviors; fuzzy action groups, i.e., a group of similar actions that the encoder struggles to distinguish, limit the classification performance of skeleton behavior recognition models. Effectively capturing the most distinctive and representative motion and semantic features between similar action classes remains a challenge.

[0004] Patent application CN114170683A discloses a behavior recognition method and system based on a multimodal attention fusion network. First, it uses the ResNeXt101 network within the multimodal attention fusion network to extract RGB features from the video to be recognized. Then, it uses the Shift-GCN network within the multimodal attention fusion network to extract skeletal features. Next, it uses the modal fusion network within the multimodal attention fusion network to perform modal fusion of the RGB and skeletal features, determining the fused features. Finally, it uses the channel fusion network within the multimodal attention fusion network to perform channel fusion of the fused features, determining the channel fused features. The losses from the RGB features, skeletal features, and channel fused features are integrated to determine the multimodal loss. The multimodal attention fusion network is then optimized using this multimodal loss to output the behavior recognition result. This application uses RGB and skeleton data for multimodal feature fusion and employs the GCN method for action recognition and classification. However, this method requires extensive labeling of the dataset, and using RGB image data presents challenges such as high storage space requirements and susceptibility to complex environmental influences, which are detrimental to improving recognition accuracy. Summary of the Invention

[0005] To overcome the problems of the prior art, the present invention aims to provide a self-supervised skeleton behavior recognition method, system, device, and medium based on action semantic guidance. A novel contrastive learning framework is developed by combining multimodal training. It uses automatically generated prototype prompts for skeleton action recognition and a large-scale pre-trained language model (GPT-3) as a knowledge engine to generate a set of related meaningful action text descriptions for each type of action. The text descriptions of actions are used to add additional action semantic information to positive and negative sample pairs, assisting the model in learning better action representations and improving the accuracy and robustness of the action recognition model.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A self-supervised skeleton behavior recognition method based on action semantic guidance is proposed. First, training skeleton video samples are acquired and preprocessed. Second, the action semantic generation module generates semantic prototypes for each type of action in the preprocessed human skeleton dataset, thus generating corresponding action text semantic features. Third, the preprocessed human skeleton dataset is input into a feature enhancement module (FAM) to enhance the skeleton data and generate corresponding action skeleton features. Finally, a multimodal fusion module is used to fuse the action text semantic features and skeleton features of the corresponding actions, assisting the model in learning more distinctive and representative action representations from a semantic level, improving the accuracy of fuzzy action recognition. Through contrastive learning, a self-supervised approach is adopted to reduce the dependence on large-scale labeled data while ensuring model recognition performance, achieving skeleton-based human action classification.

[0008] A self-supervised skeleton behavior recognition method based on action semantic guidance specifically includes the following steps:

[0009] Step 1, Collect training skeleton video samples: Use public network resources to obtain a dataset containing human skeleton movements and perform preprocessing operations, including extracting raw skeleton data, removing noise interference and viewpoint normalization, to obtain a preprocessed human skeleton dataset.

[0010] Step 2: Generate semantic prototypes for each type of action in the preprocessed human skeleton dataset through the action semantic generation module. Specifically, firstly, action prompts are designed for Large Scale Language Model (LLM) to generate behavioral descriptions, and secondly, the generated descriptions are fed into the text encoder to generate corresponding action text semantic features.

[0011] Step 3: Input the preprocessed human skeleton dataset from Step 1 into the Feature Enhancement (FAM) module to enhance the skeleton data; the enhanced skeleton data is then processed by the skeleton encoder, projected, and embedded to generate skeleton features corresponding to the action.

[0012] Step 4: Input the semantic features of the action text obtained in Step 2 and the skeleton features obtained in Step 3 into the multimodal feature fusion module (FAM) to achieve the fusion of semantic features and skeleton features; By comparing the setting of the loss function, compare the similarity of the fused feature representations generated by different action samples in the same feature space to distinguish different action samples, and backpropagate to train the skeleton encoder; Finally, add a fully connected layer with a Softmax activation function after the skeleton encoder trained by backpropagation, and use a fully supervised method to train the classifier to achieve action classification.

[0013] Step 2 includes:

[0014] First, a set of action text descriptions for each type of action in the dataset is generated from three levels: paragraph, synonym, and contextual inference. Then, a clustering algorithm is used to obtain the semantic prototype representation of the action. Subsequently, the semantic prototype representation is input into the text encoder E. t In (·), a semantic prototype vector is generated:

[0015]

[0016] Where Text represents the action description generated by LLM, TextEncoder(·) represents the text encoder, Normalization(·) represents the normalization of aggregated features, and the projection of the generated normalized features. It is a feature vector that embeds text information.

[0017] The Feature Enhancement Module (FAM) described in step 3 includes a data enhancement (including cropping and shearing) module and a random skeleton mask data enhancement module;

[0018] 3.1 In the data augmentation module, given the original skeleton sequence S, the augmented skeleton sequence is obtained through pruning and shearing transformations, using a query encoder. Key-value encoder for momentum updates Spatiotemporal features of the human skeleton are extracted, and then global average pooling (GAP) is used to obtain the query embedding z and key-value embedding.

[0019] 3.2 Through the random skeleton mask data augmentation module, action recognition is performed on a partial sequence within the temporal action interval:

[0020] 3.2.1 Visualize the original skeleton sequence. Since actions typically occur within a small local temporal region, there is no motion in the first few time frames of the action. Represent the enhanced skeleton sequence and the adjacency matrix representing the physical connections of human joints in the data augmentation module as X and A, respectively. Feature extraction is calculated as follows:

[0021]

[0022] here, It is an angle matrix, where I is the identity matrix, σ(·) is the activation function, W is a learnable matrix, and a skeleton encoder is used as the input enhanced skeleton sequence;

[0023] 3.2.2 Features X of the enhanced skeleton sequence described above a Truncate the sequence by c%, discard the remaining sequence, and generate a dense set of static anchors (StaticAncor):

[0024]

[0025] Here, is the data of the t-th frame in the i-th batch of sequences, where t is within the truncated range, i.e., 0 < t < T×c%, N is the batch size, V is the number of nodes, and c is a hyperparameter;

[0026] 3.2.3 Calculate the original skeleton S n,1 , and the similarity with the static anchor S n,2 To mine the motion area, the static anchor is compared with the skeleton sequence obtained in the data augmentation module to generate a mask matrix M r :

[0027]

[0028] Here, X ∈ R C×T×V is the original skeleton data, A ∈ R V×V , n is the number of skeleton sequences, f is the skeleton encoder, T is the data augmentation, c is a hyperparameter, and M r ∈ R C×T×V is a [0,1] mask matrix;

[0029] 3.2.4 Mask some time frame data according to the mask matrix to generate a masked skeleton sequence gr, which is expressed as:

[0030]

[0031] The masked sequence needs to be unified into a sequence data of a fixed length, and the masked skeleton sequence is adjusted to the target length by multiple interpolations.

[0032] The specific method of step 4 is as follows:

[0033] 4.1 First, fuse the feature vector after data augmentation obtained in step 3.1 with the text feature vector obtained in step 2 to obtain the corresponding fused multi-modal feature vector:

[0034]

[0035] Among them, z' respectively represent the multi-modal feature vector obtained by fusing the feature vector after data augmentation of cropping and shearing transformation with the text feature vector, linear(·) represents a linear transformation, and Alignment fuses the skeleton feature and the text feature; define a first-in-first-out memory queue The embedding m in Q i is the key-value embedding z' of the previous round, where K represents the queue size, and z' form a positive sample pair, while z' and the embedding m in Q i form a negative sample pair;

[0036] InfoNCE loss can be expressed as:

[0037]

[0038] in, This represents the similarity between the embedding z' and the memory queue Q;

[0039] 4.2 Similarly, the skeleton sequence g after masking r The query embedding is obtained after Global Average Pooling (GAP). and z and z' form two positive sample pairs, which will be used below. represent and z', and The embeddings in Q form negative sample pairs; therefore, the masked view The contrast loss is expressed as:

[0040]

[0041] here, To query the encoder, its parameters are updated via gradient backpropagation. For key-value encoders, and The parameters use a query encoder. The moving average update is represented as:

[0042]

[0043] Where m∈[0,1) is a momentum coefficient, usually close to 1, to maintain consistency of embedding in the memory queue;

[0044] Finally, optimize the encoder using the loss L:

[0045]

[0046] Where λ is a hyperparameter that balances two contrast pairs;

[0047] The encoder is updated using the gradient descent algorithm based on the calculated loss function L. The parameters are used to iterate and train the model until the set number of training rounds is reached, while retaining the encoder. It serves as a feature extractor for subsequent downstream tasks, enabling action classification.

[0048] A self-supervised skeleton behavior recognition system based on action semantic guidance includes:

[0049] The action semantic generation module is used in step 2. It realizes the semantic prototype representation of a set of action text descriptions at three levels: paragraph, synonym and context reasoning for each type of action through a clustering algorithm. Then, it generates behavior descriptions by carefully designing action cues for LLMs. The generated descriptions are fed into the text encoder to generate corresponding action text semantic features.

[0050] The Feature Enhancement Module (FAM) is used in step 3. First, it performs conventional enhancement of the skeleton sequence through pruning and shearing transformation. Second, it sets static anchor points, obtains a mask matrix by comparing it with the skeleton sequence of conventional data enhancement, and implements random skeleton mask data enhancement code based on the mask matrix to realize the mining of motion regions.

[0051] The action semantic-guided multimodal fusion module is used in step 4. By superimposing the action semantic prototype vector onto the skeleton vector, it achieves the fusion of skeleton and text features, assisting the model in learning more distinctive and representative action representations from a semantic level, and improving the accuracy of fuzzy action recognition.

[0052] A self-supervised skeleton behavior recognition device based on action semantic guidance, comprising:

[0053] Memory, used to store computer programs;

[0054] A processor is used to implement the self-supervised skeleton behavior recognition method based on action semantic guidance described in steps 1 to 4 when executing the computer program.

[0055] A computer-readable storage medium storing a computer program, which, when executed by a processor, enables self-supervised skeleton behavior recognition based on action semantic guidance according to the method described in steps 1 to 4.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] 1. Action Semantic Prototype Extraction: This invention uses automatically generated prototype prompts for skeleton-based action recognition. It generates multi-faceted action text descriptions using GPT-3.5 and uses a clustering algorithm to obtain the semantic prototype representation of each action category. The text descriptions of actions are used to add additional semantic information to positive and negative sample pairs, assisting the model in learning better action representations for action recognition.

[0058] 2. Random Skeleton Mask Data Augmentation Module: This invention applies a set of data augmentation T to the original skeleton sequence. The augmented skeleton sequence is then processed by a single-layer skeleton encoder to extract skeleton spatial features, thereby increasing the difference in skeleton features within corresponding frames. By truncating several time frames from the start of the current batch of actions to generate a dense set of static anchor points, the motion region is extracted by comparing the augmented skeleton sequence with the static anchor points. This is an effective skeleton data augmentation strategy that improves the encoder's feature extraction capability.

[0059] 3. Semantic Prototype-Guided Contrastive Learning (SG-CLR) framework for skeleton-based action recognition: A novel contrastive learning framework developed in conjunction with multimodal training is used to add extra semantic information to positive and negative sample pairs using textual descriptions of actions, enhancing the model's semantic perception capabilities and assisting the model in distinguishing ambiguous actions at the semantic level.

[0060] 4. Validation Experiments and Performance Advantages: Extensive ablation experiments demonstrated the effectiveness of the proposed innovative submodule, validating the feasibility and validity of the invention. Comparative experiments with existing GCN-based and Transformer-based methods showed performance advantages on three different datasets, indicating that the SG-CLR model achieves better results in action classification.

[0061] This invention introduces an innovative skeleton action recognition method, which improves the accuracy and robustness of action classification through semantic feature extraction and multimodal feature fusion.

[0062] Compared with the prior art, the present invention has the following advantages:

[0063] 1. Effective Solution for Similarity Action Recognition: Compared to an existing action recognition method based on a multimodal attention fusion network (CN113761975B) that uses RGB and skeleton information, this invention employs semantic and skeleton information, focusing more on handling the problem of fuzzy action recognition. This is one of the key challenges of traditional skeleton action recognition methods. This invention uses multimodal data and incorporates semantic information into skeleton features to address these issues. When processing similar actions, traditional methods may struggle to distinguish between them due to subtle differences, leading to decreased recognition accuracy. However, through semantic feature extraction, the model can capture deeper and more abstract action features, which are crucial for distinguishing similar actions.

[0064] 2. Lower Data Dependence: Utilizing deep learning and multimodal information fusion methods allows for the learning of effective feature representations even in the presence of noisy data, reducing reliance on preprocessed data. Multimodal feature fusion methods integrate information from different sources, increasing the model's overall understanding of actions. This also enables the model to maintain high recognition performance when faced with action variations arising from different scenes, angles, or actors. This generalization ability is essential for handling action recognition problems in practical applications.

[0065] 3. Improved Capability for Handling Complex Actions: Semantic feature extraction focuses on understanding the meaning and context of actions. It goes beyond the physical manifestation of an action (such as the movement path of a hand) to further explore its inherent meaning (e.g., waving means greeting). This semantic understanding is particularly important for complex actions, as they often contain more abstract meanings and subtle variations, making accurate recognition difficult solely based on physical features. Semantic feature extraction can identify subtle differences in actions, effectively distinguishing very similar actions and improving the recognition rate of fine-grained movements. In dynamic or complex actions, multimodal fusion can effectively utilize various information sources while maintaining recognition performance.

[0066] 4. Improved Data Efficiency: Contrastive learning effectively utilizes unlabeled data, generating a large number of training samples through self-supervision. This is particularly valuable for action recognition tasks, as collecting and labeling large amounts of high-quality action data can be very time-consuming and expensive. Secondly, semantic information only needs to be added during the training phase, and only after skeleton data classification is completed using self-supervised contrastive learning, thus saving on the cost of labeling datasets. By performing contrastive learning on multimodal features, more value can be extracted from existing data, improving data utilization efficiency.

[0067] 5. Broad Application Potential: By leveraging semantic features and multimodal information, the model can learn more generalized action representations. These representations are not limited to action instances appearing in the training data but capture the essential features of actions. Contrastive learning, by training the model with a large number of positive and negative sample pairs, helps the model capture more general data features, rather than simply overfitting to specific attributes of the training data. This means that models trained using contrastive learning methods have better generalization ability and can perform better on unseen data. Given the excellent robustness and generalization ability of this invention, it has broad application potential, including motion analysis, health monitoring, virtual reality, pose recognition, and security fields, with promising application prospects.

[0068] This invention focuses on a specially designed model. First, feature extraction is performed on both skeleton and text data. For skeleton data, random skeleton masking is used to augment the data and obtain richer hard samples. ST-GCN is used as the encoder to capture the dependencies between human joints. For text data, a pre-trained language model (GPT-3.5) is used to extract semantic features of multi-level action descriptions, and clustering is used to extract semantic prototypes to guide subsequent learning. Finally, the semantic vectors are concatenated into the skeleton feature vectors to generate a unified multimodal feature representation. Using the fused multimodal features, the model is trained through contrastive learning to optimize its ability to distinguish between similar actions.

[0069] Overall, this invention, through its combination of semantic prototype guidance, large-scale pre-trained language models, and contrastive learning, enables the understanding and recognition of human actions at different levels, demonstrating higher accuracy and efficiency, especially when dealing with complex and subtle differences in actions. This method is not only theoretically universal but also exhibits significant performance advantages in practical applications.

[0070] In summary, this invention employs a multimodal mechanism, a design that demonstrates universality across different datasets without requiring personalized adjustments for each dataset. By focusing attention on the focal motion frame, it not only more effectively recognizes human actions but also automatically adapts to feature differences across datasets. Compared to traditional data augmentation methods, our model prioritizes learning key information from motion frame data. By utilizing a semantically guided multimodal fusion module, it solves the problem of distinguishing similar actions, improves action recognition capabilities in real-world scenarios, and achieves high-precision skeleton action classification results, providing an efficient and reliable solution for applications in multiple fields. Attached Figure Description

[0071] Figure 1 Flowchart for the skeleton motion classification method.

[0072] Figure 2 This is a diagram of the overall architecture of a semantically guided skeleton behavior recognition model.

[0073] Figure 3 A framework diagram generated for the action description prototype.

[0074] Figure 4 A graph illustrating strategies for enhancing random skeleton mask data. Detailed Implementation

[0075] This embodiment presents a self-supervised skeleton behavior recognition method based on action semantic guidance. The process is as follows: Figure 1 The specific network structure framework it adopts is as follows: Figure 3 As shown:

[0076] Step 1, Collect skeleton video samples to be trained: Use public network resources to obtain a dataset containing human skeleton movements and perform preprocessing operations, including extracting the original skeleton data, removing noise interference and viewpoint normalization, to obtain a preprocessed dataset.

[0077] Step 2 involves designing action cues for a large-scale language model (LLM) to generate richer, more granular behavioral descriptions. These descriptions are then fed into a text encoder to generate corresponding action text semantic features. The specific method is as follows:

[0078] First, the generative capabilities of GPT-3.5 are utilized to generate a set of relevant and meaningful action text descriptions at three levels: paragraph, synonym, and contextual inference description. Then, a semantic prototype representation of each action category is obtained using the K-means clustering algorithm. Finally, the generated final descriptions are input into the text encoder E. t In (·), a semantic prototype vector is generated.

[0079]

[0080] Where Text represents the action description generated by LLM, TextEncoder(·) represents the text encoder, Normalization(·) represents the normalization of aggregated features, and the projection of the generated normalized features. It is a feature vector that embeds text information.

[0081] Step 3: Input the preprocessed human skeleton dataset from Step 1 into the Feature Enhancement (FAM) module to enhance the skeleton data. The enhanced skeleton sequence is processed by a skeleton encoder, projected, and embedded to generate skeleton features corresponding to the action; the specific method is as follows:

[0082] 3.1 In the normal data augmentation module, given the original skeleton sequence S, the augmented skeleton sequence is obtained through cropping and shearing, and then a query encoder is used. Key-value encoder for momentum updates Spatiotemporal features of the human skeleton are extracted, and then global average pooling (GAP) is used to obtain the query embedding z and key-value embedding.

[0083] 3.2 such as Figure 4As shown, according to the random skeleton mask data augmentation module, action recognition is usually performed on partial sequences in the temporal action interval, and the information shared in the temporal partial sequences is more discriminative for action recognition.

[0084] 3.2.1 First, visualize the original skeleton sequence. Since actions usually occur in a very small local temporal region, there is usually no movement in the first few time frames when the action starts. The augmented skeleton sequence and the corresponding adjacency matrix are denoted as X and A respectively, and feature extraction can be calculated as:

[0085]

[0086] Here and I are the angular matrix and the identity matrix respectively, σ(·) is the activation function. W is the learnable matrix. A single-layer spatial graph convolutional network in ST-GCN is used as the skeleton encoder, and the input is the augmented skeleton sequence.

[0087] 3.2.2 Then, truncate c% of the feature-enhanced skeleton sequence X a and discard the remaining sequence to generate a set of dense static anchors (StaticAncor), which can be calculated as:

[0088]

[0089] Here is the data of the t-th frame in the i-th batch of sequences, where t is within the range of our truncation, i.e., 0 < t < T×c%. N is the batch size, V is the number of nodes, and c is the hyperparameter.

[0090] 3.2.3 Calculate the similarity between the original skeleton S n,1 , and the static anchor S n,2 . To挖掘运动区域, by comparing the static anchor with the skeleton sequence obtained by normal augmentation, a mask matrix M r is generated, which can be expressed as:

[0091]

[0092] Here X ∈ R C×T×V is the original skeleton data, A ∈ R V×V , n is the number of skeleton sequences, f is the skeleton encoder, T is the data augmentation, c is the hyperparameter, and M r ∈ R C×T×V is a [0,1] mask matrix.

[0093] 3.2.4 Finally, generate a masked skeleton sequence g r , which can be expressed as:

[0094]

[0095] The masked sequence needs to be unified into a fixed-length sequence of data, so cubic interpolation is then used to fill the entire skeleton sequence.

[0096] Step 4: Input the semantic features of the action text obtained in Step 2 and the skeleton features obtained in Step 3 into the multimodal feature fusion module to achieve the fusion of semantic features and skeleton features. Different action samples are distinguished by comparing the similarity of the feature representations generated by different action samples in the same feature space, and the encoder is trained through backpropagation. Finally, a fully connected layer with a Softmax activation function is added to the pre-trained encoder with fixed parameters, and a fully supervised method is used to train the classifier to achieve action classification. The specific method is as follows:

[0097] 4.1 First, the data-enhanced feature vector obtained in step 3 is fused with the text feature vector obtained in step 2 to obtain the fused multimodal feature vector, which can be expressed as:

[0098]

[0099] Here, linear(·) represents a linear transformation, and Alignment fuses skeleton features with text features. Key value z and z' are stored in a first-in-first-out memory queue. In the context of queue size, K represents the queue size. z' and its embedding in Q form positive sample pairs, while z' and its embedding in Q form negative sample pairs. The InfoNCE loss can be expressed as:

[0100]

[0101] in, This indicates the similarity between the embedding of view v and the memory queue Q.

[0102] 4.2 Similarly, mask embedding g r With the corresponding key-value embedding g +r For positive sample pairs, g here +r represent and z', while g r The embeddings in Q form negative sample pairs. Therefore, the masked view g r The contrast loss can be expressed as:

[0103]

[0104] The query encoder here The parameters of the key encoder are updated via gradient backpropagation. The parameters are updated to the moving average of the query encoder, which can be expressed as:

[0105]

[0106] Where m∈[0,1) is a momentum coefficient, usually close to 1, to maintain consistency in embedding within the memory queue.

[0107] Finally, the loss used to optimize the encoder can be expressed as:

[0108]

[0109] Here, λ is a hyperparameter that balances the two contrast pairs. The overall architecture diagram of the semantically guided skeleton behavior recognition model is shown below. Figure 2 As shown.

[0110] Experimental analysis was conducted to optimize the model hyperparameters. Three publicly available large-scale datasets were used: NTU-RGB+D, NTU-RGB+D120, and PKU-MMD. Preprocessing operations were performed on the skeleton data, including raw skeleton acquisition, noise removal, and viewpoint normalization.

[0111] The processed dataset is fed into a multimodal contrastive learning model based on 3D skeleton action representations. To extract high-level semantic information from action sequences, it integrates a novel enhancement strategy, SkeleMask, and a multimodal feature fusion method, SG-CL. SkeleMask enhances the original skeleton sequence from a temporal perspective, encouraging the model to build high-quality, challenging contrast pairs. These pairs help the model learn more general and discriminative representations. SG-CL uses LLMs to generate relevant action text descriptions, adding additional semantic information to positive and negative sample pairs, assisting the model in learning better action representations for action recognition. Therefore, the method of this invention significantly improves the performance of the model on downstream tasks. Extensive experimental results demonstrate the effectiveness of the SG-CLR of this invention and show that it achieves state-of-the-art performance under various evaluation protocols.

[0112] The operating system used in the experiment was Ubuntu 22.04.1, and the deep learning framework used was PyTorch. The specific configurations involved in the experiment are shown in Table 1.

[0113] Table 1 Experimental Configuration Table

[0114] project Configuration Processor (CPU) 12thGenIntel(R)Core(TM)i9-12900KFcpu@3.20GHz Graphics card (GPU) GeForce RTX 3090 operating system Ubuntu 22.04.1 frame PyTorch

[0115] Extensive ablation experiments demonstrated the effectiveness of the innovative sub-module proposed in this invention, reinforcing its feasibility and performance. Comparative experiments with existing GCN-based and Transformer-based methods showed performance advantages on three different datasets. This implies that the SG-CLR model achieves better results in action classification. Table 2 illustrates the impact of different action representation methods on model performance, Table 3 lists the impact of different action interpolation methods on model performance, and Table 4 presents the linear evaluation results of the model of this invention compared to state-of-the-art methods on multiple action recognition benchmarks.

[0116]

[0117] Table 4: Linear evaluation results for different modalities under different settings of the human skeleton datasets NTU-60, NTU-120, and PKU-MMD. J, B, and M represent skeleton flow, skeletal flow, and motion flow, respectively. J+B+M represents the fusion of the three flows.

[0118]

Claims

1. A self-supervised skeleton behavior recognition method based on action semantic guidance, characterized in that, First, obtain the skeleton video sample to be trained and pre-process it; Second, generate the semantic prototype of each type of action in the pre-processed human skeleton data set through an action semantic generation module, generate the corresponding action text semantic features, specifically, first design an action prompt for a large language model (LLM) to generate a behavior description, and then the generated description is fed into a text encoder to generate the corresponding action text semantic features; In addition, input the pre-processed human skeleton data set into a feature enhancement module (FAM), which includes a data enhancement module and a random skeleton mask data enhancement module; In the data augmentation module, given the original skeleton sequence S, the augmented skeleton sequence is obtained through cropping and shear transformation, and the query encoder and momentum updated key-value encoder The spatiotemporal features of the human skeleton are extracted, and then the global average pooling GAP is used to obtain the query embedding z and the key-value embedding ; Through the random skeleton mask data enhancement module, the action recognition is performed on part of the sequence in the time sequence action interval: Visualize the original skeleton sequence, as actions usually occur in a very small local time sequence area, and no movement occurs in the first few time frames of the action start; the enhanced skeleton sequence in the data enhancement module and the adjacency matrix representing the physical connection of human joints are represented as X and A respectively, and the feature extraction calculation is: Here, = A+I, is a diagonal matrix, I is an identity matrix, (·) is an activation function; is a learnable matrix; adopts a skeleton encoder to input the enhanced skeleton sequence; Features of the enhanced skeleton sequence described above Truncate c%, discard the rest of the sequence, generate a set of dense static anchors Static Ancor: is the data of the t-th frame in the i-th batch of sequences, t is within the truncated range, 0 < t < T x c%, N is the batch size, V is the number of nodes, and c is a hyperparameter; Compute the original skeleton Similarity between static anchor points In order to mine the motion area, the static anchor points are compared with the skeleton sequence obtained in the data enhancement module to generate a mask matrix : X ∈ R C×T×V is the original skeleton data, A ∈ R V×V , n is the number of skeleton sequences, f is the skeleton encoder, T is data augmentation, c is a hyperparameter, M r ∈ R C×T ×V is a [0, 1] mask matrix; According to the mask matrix, some time frame data is masked to generate a masked skeleton sequence gr, represented as: The masked sequence needs to be unified into a fixed length sequence data, and the masked skeleton sequence is adjusted to the target length by multiple interpolation to realize skeleton data enhancement, and the skeleton features of the corresponding action are generated; Finally, use a multi-modal fusion module to fuse the action text semantic features and skeleton features of the corresponding action, to help the model learn more discriminative and representative action representations from the semantic level, improve the recognition accuracy of ambiguous actions, and reduce the dependence on large-scale labeled data while ensuring the recognition performance of the model through the method of contrastive learning in a self-supervised manner, to realize human action classification based on skeletons.

2. The method of claim 1, wherein the method is a motion semantic guided self-supervised skeleton action recognition method. Specifically, the following steps are included: Step 1: Collect the skeleton video sample to be trained: use public network resources to obtain a data set containing human skeleton actions, and perform pre-processing operations, including extracting original skeleton data, removing noise interference, and normalizing the view angle, to obtain a pre-processed human skeleton data set; Step 2: Generate the semantic prototype of each type of action in the pre-processed human skeleton data set through an action semantic generation module; Step 3: Input the pre-processed human skeleton data set in step 1 into a feature enhancement module (FAM) to realize skeleton data enhancement; the enhanced skeleton data is projected, embedded, and generated into skeleton features of the corresponding action through a skeleton encoder; Step 4: Input the action text semantic features obtained in step 2 and the skeleton features obtained in step 3 into a multi-modal feature fusion module to realize semantic feature and skeleton feature fusion; by setting a loss function, the similarity of the fused feature representations generated by different action samples in the same feature space is compared to distinguish different action samples, and the skeleton encoder is trained in reverse; finally, a fully connected layer with a Softmax activation function is added after the reverse propagation trained skeleton encoder, and a fully supervised method is used to train the classifier to realize action classification.

3. The action semantic guided self-supervised skeleton-based behavior recognition method of claim 2, wherein, The step 2 includes: First, a set of action text descriptions of each class of action contained in the dataset is generated from three levels of paragraph, synonym and context reasoning, and then the semantic prototype representation of the action is obtained through a clustering algorithm; subsequently, the semantic prototype representation is input into the text encoder to generate a semantic prototype vector: wherein, represents an action description generated by the LLM represents a text encoder, represents a normalization of the aggregated features, a projection of the generated normalized features is a feature vector embedding the textual information.

4. The action semantic guided self-supervised skeleton-based behavior recognition method according to claim 2, wherein, The specific method of step 4 is: 4.1 First, the feature vector after data enhancement is fused with the text feature vector obtained in step 2 to obtain a corresponding fused multi-modal feature vector: wherein, respectively represent the multi-modal feature vectors fused by the feature vectors after the cropping and the cutting transformation data enhancement and the text feature vectors, (·) represents a linear change, fuse the skeleton feature and the text feature; define a memory queue of first-in-first-out the embedding in Q is the key-value embedding of the last round wherein, K represents a queue size, ' and form a positive sample pair, while ' and the embedding in Q form a negative sample pair; The InfoNCE loss can be expressed as: wherein, represents an embedding similarity between the memory queue Q; 4.2 Masked skeleton sequence g r Query embedding is obtained by global average pooling (GAP) , and respectively constitute two positive sample pairs, and the following represents , and and the embedding in Q form a negative sample pair; therefore, the contrast loss of the masked view is expressed as: is a query encoder whose parameters are updated by gradient backpropagation, is a key-value encoder, and parameters of are updated using the query encoder moving average updates of are denoted as: Where m [0, 1) is a momentum coefficient close to 1 to maintain the consistency of the embedding in the memory queue; Finally, the encoder is optimized with the loss L: Where λ is a hyperparameter that balances the two contrast pairs; The parameters of the encoder are updated according to the calculated loss function L by a gradient descent algorithm , and iterative training is performed until the model reaches a set number of training rounds, and the encoder is retained as a feature extractor for subsequent downstream tasks to implement action classification.

5. A self-supervised skeleton action recognition system based on action semantic guidance, for implementing the method of claim 2, characterized in that, Including: The action semantic generation module is used in step 2 to realize the semantic prototype representation of a set of action text descriptions in three levels of paragraphs, synonyms and context reasoning for each class of actions through clustering algorithm, and then generate behavior description through carefully designed action prompts for LLMs. The generated description is fed into the text encoder to generate corresponding action text semantic features; The feature enhancement module FAM is used in step 3. First, the regular enhancement of the skeleton sequence is realized through cropping and shear transformation; secondly, the static anchor points are set, and the mask matrix is obtained by comparing with the regular data enhanced skeleton sequence, and the random skeleton mask data enhancement is realized according to the mask matrix to realize the mining of the motion area; The action semantic guided multi-modal fusion module is used in step 4 to realize the fusion of skeleton and text features by superimposing action semantic prototype vectors and skeleton vectors, to help the model learn more distinctive and representative action representations from the semantic level, and to improve the recognition accuracy of fuzzy actions.

6. A motion semantic-guided self-supervised skeleton action recognition device, characterized in that, Including: Memory for storing computer programs; The processor is used to execute the computer program to realize the action semantic guided self-supervised skeleton behavior recognition method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: receiving a request for a resource from a client; determining whether the client is authorized to access the resource; and if the client is authorized to access the resource, providing the resource to the client. The computer program can be executed by the processor to realize the action semantic guided self-supervised skeleton behavior recognition based on the method of any one of claims 1 to 4.

Images