Self-supervised skeleton behavior recognition method, system and equipment based on action semantic guidance and medium

Abstract

The invention discloses a self-supervised skeleton behavior recognition method, system and device based on action semantic guidance and a medium, and the method comprises the steps: firstly, obtaining a to-be-trained skeleton video sample, and carrying out the preprocessing of the to-be-trained skeleton video sample; secondly, generating a semantic prototype of each type of action in the preprocessed human body skeleton data set through an action semantic generation module, namely generating a corresponding action text semantic feature; besides, the preprocessed human body skeleton data set is input into a feature enhancement module (FAM), skeleton data enhancement is achieved, and skeleton features corresponding to actions are generated; and finally, fusing action text semantic features and skeleton features of corresponding actions by using a multi-modal fusion module, assisting the model in learning more distinctive and representative action representations from the semantic level, improving the fuzzy action recognition precision, and through a comparative learning method, improving the fuzzy action recognition accuracy. A self-supervision mode aims to guarantee model recognition performance and reduce dependence on large-scale labeled data at the same time, and skeleton-based human body action classification is achieved; the system, the device and the medium are based on a self-supervised skeleton behavior recognition method guided by action semantics, human body actions are recognized and classified, and the system, the device and the medium have the advantages of being high in accuracy and good in robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of human motion recognition based on skeletons, and in particular to a self-supervised skeleton behavior recognition method, system, device and medium based on action semantic guidance. Background Art

[0002] Actions and behaviors belong to the category of human biological codes. Identifying and understanding the actions and behaviors of the observed individuals is a basic psychological attribute of human visual perception and cognition. As a basic and important topic in computer vision, human action recognition has a wide range of applications, including video retrieval, human-computer interaction, sign language recognition, and autonomous driving. In addition, behavior recognition plays a vital role in individual identification, communication interaction, and inferring the behavioral intentions of others.

[0003] With the rapid development of depth sensors and pose estimation algorithms, 3D skeleton data has become easier to obtain. In addition, unlike other modal data, compact skeleton data provides specific position and motion information of human joints, facilitates the construction of space-time and motion, avoids the interference of background environment, and can effectively improve the accuracy and robustness of action recognition in noisy scenes. In the past few years, self-supervised contrastive learning skeleton action recognition methods have achieved excellent performance. However, these methods still face many problems: first, the existing skeleton data enhancement strategies cannot provide rich difficult contrast pairs for contrastive learning to effectively learn general discriminative spatiotemporal skeleton motion representations; second, skeleton data lacks contextual information and interactive objects, and the difference between similar action classes is small. Therefore, direct modeling based on skeleton spatiotemporal features brings limited returns in terms of performance for fuzzy behaviors that are difficult to distinguish; fuzzy action groups, i.e., a group of similar actions that are difficult for the encoder to distinguish, limit the classification performance of skeleton action recognition models. How to effectively capture the most distinctive and representative motion features and semantic features between similar action classes remains challenging.

[0004] The patent application is a behavior recognition method and system based on a multimodal attention fusion network (CN114170683A). First, the ResNeXt101 network in the multimodal attention fusion network is used to extract the RGB features in the video to be recognized; and the Shift-GCN network in the multimodal attention fusion network is used to extract the skeleton features; the modal fusion network in the multimodal attention fusion network is used to perform modal fusion on the RGB features and the skeleton features to determine the features after modal fusion; the channel fusion network in the multimodal attention fusion network is used to perform channel fusion on the features after modal fusion to determine the features after channel fusion; the loss of RGB features, the loss of skeleton features and the loss of features after channel fusion are integrated to determine the multimodal loss, and the multimodal loss is used to optimize the multimodal attention fusion network to output the behavior recognition result. This application uses RGB data and skeleton data for multimodal feature fusion, and uses the GCN method for action recognition classification. This method requires a lot of labeling work on the data set, and the use of RGB image data faces the problem of occupying a large amount of storage space and being easily affected by complex environments, which is not conducive to improving recognition accuracy. Summary of the invention

[0005] In order to overcome the problems of the above-mentioned prior art, the purpose of the present invention is to provide a self-supervised skeleton behavior recognition method, system, device and medium based on action semantics guidance, and develop a new contrastive learning framework in combination with multimodal training, which uses automatically generated prototype prompts for skeleton action recognition, and uses a large-scale pre-trained language model (GPT-3) as a knowledge engine to generate a set of relevant and meaningful action text descriptions for each type of action, and uses the text description of the action to add additional action semantic information to the positive and negative sample pairs, to assist the model in learning better action representation, and to improve the accuracy and robustness of the action recognition model.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A self-supervised skeleton action recognition method based on action semantics guidance is proposed. First, the skeleton video samples to be trained are obtained and preprocessed. Secondly, the semantic prototype of each type of action in the preprocessed human skeleton dataset is generated through the action semantic generation module, that is, the corresponding action text semantic features are generated. In addition, the preprocessed human skeleton dataset is input into the feature enhancement module (FAM) to realize skeleton data enhancement and generate skeleton features of the corresponding action. Finally, the multimodal fusion module is used to fuse the action text semantic features and skeleton features of the corresponding action, so as to assist the model to learn more distinctive and representative action representations from the semantic level and improve the accuracy of fuzzy action recognition. Through the comparative learning method, the skeleton-based human action classification is realized in a self-supervised manner while ensuring the recognition performance of the model and reducing the dependence on large-scale labeled data.

[0008] A self-supervised skeleton behavior recognition method based on action semantics guidance specifically includes the following steps:

[0009] Step 1, collecting skeleton video samples to be trained: using public network resources to obtain a data set containing human skeleton movements, and performing a preprocessing operation, wherein the preprocessing operation includes extracting original skeleton data, removing noise interference and normalizing the view angle to obtain a preprocessed human skeleton data set;

[0010] Step 2: Generate semantic prototypes for each type of action in the preprocessed human skeleton dataset through the action semantic generation module. Specifically, firstly, design action prompts for the large-scale language model (LLM) to generate behavior descriptions. Then, the generated descriptions are fed to the text encoder to generate the corresponding action text semantic features.

[0011] Step 3, inputting the human skeleton data set preprocessed in step 1 into the feature enhancement module (FAM) to enhance the skeleton data; the enhanced skeleton data is projected and embedded through the skeleton encoder to generate skeleton features of the corresponding action;

[0012] In step 4, the action text semantic features obtained in step 2 and the skeleton features obtained in step 3 are input into the multimodal feature fusion module (FAM) to achieve the fusion of semantic features and skeleton features; by comparing the setting of the loss function, the similarity of the fused feature representations generated by different action samples in the same feature space is compared to distinguish different action samples, and the skeleton encoder is trained by back propagation; finally, a fully connected layer with a Softmax activation function is added after the skeleton encoder trained by back propagation, and a fully supervised method is used to train the classifier to achieve action classification.

[0013] The step 2 comprises:

[0014] First, a set of action text descriptions for each type of action in the dataset is generated from the three levels of paragraph, synonym and context reasoning. Then, the semantic prototype representation of the action is obtained through clustering algorithm. Then, the semantic prototype representation is input into the text encoder E t (·), generate the semantic prototype vector:

[0015]

[0016] Among them, Text represents the action description generated by LLM, TextEncoder(·) represents the text encoder, Normalization(·) represents the normalization of the aggregated features, and the projection of the generated normalized features is the feature vector that embeds the text information.

[0017] The feature enhancement module (FAM) described in step 3 includes a data enhancement (including crop and shear) module and a random skeleton mask data enhancement module;

[0018] 3.1 In the data enhancement module, given the original skeleton sequence S, the enhanced skeleton sequence is obtained by cropping and shearing transformation, and the query encoder is used and momentum updated key-value encoder Extract the spatiotemporal features of the human skeleton, and then use global average pooling (GAP) to obtain the query embedding z and key value embedding

[0019] 3.2 Through the random skeleton mask data enhancement module, action recognition is performed on partial sequences of temporal action intervals:

[0020] 3.2.1 Visualize the original skeleton sequence. Since the action usually occurs in a very small local temporal region, there is no movement in the first few time frames of the action. The enhanced skeleton sequence in the data enhancement module and the adjacency matrix representing the physical connection of the human body joints are represented as X and A respectively. The feature extraction calculation is:

[0021]

[0022] here, is the angle matrix, I is the unit matrix, σ(·) is the activation function; W is the learnable matrix; a skeleton encoder is used to input the enhanced skeleton sequence;

[0023] 3.2.2 Feature X of the enhanced skeleton sequence a Perform a truncation c%, discard the remaining sequences, and generate a set of dense static anchors (StaticAncor):

[0024]

[0025] Here, is the data of the t-th frame in the i-th batch of sequences, where t is within the truncated range, i.e., 0 < t < T×c%, N is the batch size, V is the number of nodes, and c is a hyperparameter;

[0026] 3.2.3 Calculate the original skeleton S n,1 , and the similarity with the static anchor S n,2 To mine the motion area, the static anchor is compared with the skeleton sequence obtained in the data augmentation module to generate a mask matrix M r :

[0027]

[0028] Here, X ∈ R C×T×V is the original skeleton data, A ∈ R V×V , n is the number of skeleton sequences, f is the skeleton encoder, T is the data augmentation, c is a hyperparameter, and M r ∈ R C×T×V is a [0,1] mask matrix;

[0029] 3.2.4 Mask some time frame data according to the mask matrix to generate a masked skeleton sequence gr, which is expressed as:

[0030]

[0031] The masked sequence needs to be unified into a sequence data of a fixed length, and the masked skeleton sequence is adjusted to the target length by multiple interpolations.

[0032] The specific method of step 4 is as follows:

[0033] 4.1 First, fuse the feature vector after data augmentation obtained in step 3.1 with the text feature vector obtained in step 2 to obtain the corresponding fused multi-modal feature vector:

[0034]

[0035] Among them, z' respectively represent the multi-modal feature vector obtained by fusing the feature vector after data augmentation of cropping and shearing transformation with the text feature vector, linear(·) represents a linear transformation, and Alignment fuses the skeleton feature with the text feature; define a first-in-first-out memory queue The embedding m in Q i is the key-value embedding z' of the previous round, where K represents the queue size, and z' form a positive sample pair, while z' and the embedding m in Q i form a negative sample pair;

[0036] InfoNCE loss can be expressed as:

[0037]

[0038] in, represents the similarity between the embedding z' and the memory queue Q;

[0039] 4.2 Similarly, the masked skeleton sequence g r The query embedding is obtained through global average pooling (GAP) and and z' respectively constitute two positive sample pairs, and the following uses represent and z', while and the embedding in Q form a negative sample pair; therefore, the mask view The contrast loss is expressed as:

[0040]

[0041] here, To query the encoder, its parameters are updated via gradient backpropagation, is a key-value encoder, and The parameters of the query encoder are used The moving average update is expressed as:

[0042]

[0043] Where m∈[0,1) is a momentum coefficient, which is usually close to 1 to maintain the consistency of embedding in the memory queue;

[0044] Finally, the encoder is optimized with loss L:

[0045]

[0046] Among them, λ is a hyperparameter that balances the two contrast pairs;

[0047] Update the encoder according to the calculated loss function L through the gradient descent algorithm Parameters, iterative training is performed until the model reaches the set number of training rounds, and the encoder is retained As a feature extractor, it is used in subsequent downstream tasks to achieve action classification.

[0048] A self-supervised skeleton action recognition system based on action semantics guidance, comprising:

[0049] The action semantic generation module is used in step 2. It uses a clustering algorithm to implement a set of semantic prototype representations of action text descriptions at three levels: paragraph, synonym, and context reasoning for each type of action. Then, it generates behavior descriptions by carefully designing action prompts for LLMs. The generated descriptions are fed into the text encoder to generate corresponding action text semantic features.

[0050] Feature Enhancement Module (FAM), used in step 3, firstly, implements conventional enhancement of skeleton sequence through cropping and shearing transformation; secondly, sets static anchor points, obtains mask matrix by comparing with skeleton sequence of conventional data enhancement, implements random skeleton mask data enhancement code according to mask matrix, and realizes mining of motion area;

[0051] The action semantics-guided multimodal fusion module is used in step 4 to achieve skeleton and text feature fusion by superimposing the action semantic prototype vector on the skeleton vector, assisting the model to learn more distinctive and representative action representations from the semantic level, and improving the accuracy of fuzzy action recognition.

[0052] A self-supervised skeleton behavior recognition device based on action semantics guidance, comprising:

[0053] Memory for storing computer programs;

[0054] A processor is used to implement the self-supervised skeleton behavior recognition method based on action semantics guidance described in steps 1 to 4 when executing the computer program.

[0055] A computer-readable storage medium stores a computer program, which, when executed by a processor, can implement self-supervised skeleton behavior recognition based on action semantics guidance based on the method described in steps 1 to 4.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. Action semantic prototype extraction: This paper uses automatically generated prototype prompts to perform skeleton-based action recognition. It generates multi-faceted action text descriptions through GPT-3.5 and uses clustering algorithms to obtain semantic prototype representations of each type of action. The text description of the action is used to add additional semantic information to the positive and negative sample pairs, assisting the model to learn better action representations for action recognition.

[0058] 2. Random skeleton mask data enhancement module: The present invention applies a set of data enhancement T to the original skeleton sequence. The enhanced skeleton sequence is passed through a single-layer skeleton encoder to extract skeleton spatial features to increase the skeleton feature differences in the corresponding frames. A set of dense static anchor points is generated by truncating several time frames at the beginning of the current batch of actions, and the enhanced skeleton sequence is compared with the static anchor points to mine the motion area. This is an effective skeleton data enhancement strategy that improves the feature extraction capability of the encoder.

[0059] 3. Semantic prototype-guided skeleton action recognition: A new semantic representation-guided contrastive learning framework (SG-CLR) is developed for skeleton-based action recognition. A new contrastive learning framework developed in combination with multimodal training uses text descriptions of actions to add additional textual semantic information to positive and negative sample pairs, enhancing the semantic perception ability of the model and assisting the model in distinguishing ambiguous actions from a semantic level.

[0060] 4. Verification experiments and performance advantages: Through sufficient ablation experiments, the effectiveness of the proposed innovative sub-modules is proved, which verifies the feasibility and effectiveness of the invention. Through comparative experiments with existing GCN-based methods and Transformer-based methods, the present invention demonstrates performance advantages on three different datasets, which means that the SG-CLR model achieves better results in action classification.

[0061] The present invention introduces an innovative skeleton action recognition method, which improves the accuracy and robustness of action classification through the skeleton action recognition method of semantic feature extraction and multimodal feature fusion.

[0062] Compared with the prior art, the present invention has the following advantages:

[0063] 1. Effective solution to similar action recognition: Compared with the existing behavior recognition method based on multimodal attention fusion network (CN113761975B) that uses RGB information and skeleton information, the present invention uses semantic information and skeleton information, and focuses more on dealing with the problem of ambiguous action recognition. This is one of the key challenges of traditional skeleton action recognition methods, and the present invention uses multimodal data to address these problems by adding semantic information to skeleton features. When processing similar actions, traditional methods may be difficult to distinguish due to slight differences between actions, resulting in reduced recognition accuracy. Through semantic feature extraction, the model can capture deeper and more abstract action features, which are crucial for distinguishing similar actions.

[0064] 2. Lower data dependence: Deep learning and multimodal information fusion methods can learn effective feature representations in the presence of noisy data, reducing the dependence on preprocessed data. Multimodal feature fusion methods can integrate information from different sources, increase the model's overall understanding of the action, and also enable the model to maintain high recognition performance when facing action variations caused by different scenes, different angles, or different performers. This generalization ability is necessary to deal with action recognition problems in practical applications.

[0065] 3. Better processing capabilities for complex actions: Semantic feature extraction focuses on understanding the meaning and context of an action. It is not limited to the physical manifestation of the action (such as the movement path of the hand), but further explores the intrinsic meaning of the action (such as waving means greeting). This semantic level of understanding is especially important for complex actions, because complex actions often contain more abstract meanings and subtle changes, and it is difficult to accurately identify them by relying solely on physical features. Through semantic feature extraction, subtle differences in actions can be identified, thereby effectively distinguishing very similar actions and improving the recognition rate of fine-grained actions; in dynamic or complex actions, multimodal fusion can effectively utilize various information sources to maintain recognition performance.

[0066] 4. Better data efficiency: Contrastive learning can effectively utilize unlabeled data and generate a large number of training samples in a self-supervised manner. This is particularly valuable for action recognition tasks, because collecting and annotating a large amount of high-quality action data can be very time-consuming and expensive. Secondly, semantic information only needs to be added during the training phase, and semantic information is added after the skeleton data classification is completed using the self-supervised contrastive learning method, so this saves the cost of labeling the data set. By performing contrastive learning on multimodal features, more value can be mined from existing data and data utilization efficiency can be improved.

[0067] 5. Potential for wide application: By using semantic features and multimodal information, the model can learn a more generalized action representation, which is not limited to the action instances that appear in the training data, but can capture the essential characteristics of the action. Contrastive learning trains the model through a large number of positive and negative sample pairs, which helps the model capture more general data features rather than just overfitting to the specific attributes of the training data. This means that the model trained by the contrastive learning method has better generalization ability and can perform better on unseen data. In view of the good robustness and generalization ability of the present invention, it has a wide range of application potential, including motion analysis, health monitoring, virtual reality, posture recognition and security fields, and has broad application prospects.

[0068] The present invention focuses on the special design of the model. First, feature extraction is performed on the skeleton data and text data respectively. For the skeleton data, a random skeleton mask data enhancement method is adopted to obtain richer hard samples, and ST-GCN is used as an encoder to capture the dependencies between human joints; for the text data, a pre-trained language model (GPT-3.5) is used to extract the semantic features of multi-level action descriptions, and a clustering method is used to extract semantic prototypes to guide the subsequent learning process; finally, the semantic vector is added to the skeleton feature vector by concatenation to generate a unified multimodal feature representation, and the model is trained to optimize the distinction between similar actions by contrastive learning using the fused multimodal features.

[0069] In general, the present invention, through the combination of semantic prototype guidance, large-scale pre-trained language model and contrastive learning, can understand and recognize human actions from different levels, especially when dealing with complex and subtle actions, showing higher accuracy and efficiency. This method is not only universal in theory, but also shows significant performance advantages in practical applications.

[0070] In summary, the present invention adopts a multimodal mechanism, and this design shows universality in different data sets, without the need for personalized adjustments for each data set. By focusing on the focal motion frame, it can not only more effectively identify human movements, but also automatically adapt to the feature differences of different data sets. Compared with traditional data enhancement methods, our model pays more attention to learning the key information of motion frame data from the data. By adopting a semantically guided multimodal fusion module, it solves the problem of distinguishing similar actions, improves the ability to recognize actions in real-world scenes, and achieves high-precision skeleton action classification results, providing efficient and reliable solutions for applications in multiple fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Flowchart of the skeleton action classification method.

[0072] Figure 2 This is the overall architecture diagram of the semantically guided skeleton behavior recognition model.

[0073] Figure 3 Frame diagram generated for the action description prototype.

[0074] Figure 4 Figure 2. Data augmentation strategy diagram for random skeleton masking. DETAILED DESCRIPTION

[0075] This embodiment provides a self-supervised skeleton behavior recognition method based on action semantics guidance. The process is shown in Figure 1 The specific network structure framework adopted is as follows Figure 3 As shown:

[0076] Step 1, collect skeleton video samples to be trained: use public network resources to obtain a data set containing human skeleton movements, and perform preprocessing operations, the preprocessing operations include extracting original skeleton data, removing noise interference and normalizing the perspective to obtain a preprocessed data set.

[0077] Step 2: Generate richer and more fine-grained behavior descriptions by designing action prompts for the Large Scale Language Model (LLM). The generated descriptions are fed into the text encoder to generate the corresponding action text semantic features. The specific method is as follows:

[0078] First, we use the generative capabilities of GPT-3.5 to generate a set of relevant and meaningful action text descriptions at the three levels of paragraph, synonym, and contextual inference description for each type of action, and then use the K-means clustering algorithm to obtain the semantic prototype representation of each type of action. Subsequently, the generated final description is input into the text encoder E t In (·), a semantic prototype vector is generated.

[0079]

[0080] Among them, Text represents the action description generated by LLM, TextEncoder(·) represents the text encoder, Normalization(·) represents the normalization of the aggregated features, and the projection of the generated normalized features is the feature vector that embeds the text information.

[0081] Step 3: Input the human skeleton data set preprocessed in step 1 into the feature enhancement module (FAM) to enhance the skeleton data. The enhanced skeleton sequence is projected and embedded through the skeleton encoder to generate the skeleton features of the corresponding action. The specific method is:

[0082] 3.1 In the normal data enhancement module, given the original skeleton sequence S, the enhanced skeleton sequence is obtained by cropping and shearing, and the query encoder is used and momentum updated key-value encoder Extract the spatiotemporal features of the human skeleton, and then use global average pooling (GAP) to obtain the query embedding z and key value embedding

[0083] 3.2 As Figure 4As shown, according to the random skeleton mask data augmentation module, action recognition is usually performed on partial sequences in the temporal action interval, and the information shared in the temporal partial sequences is more discriminative for action recognition.

[0084] 3.2.1 First, visualize the original skeleton sequence. Since actions usually occur in a very small local temporal region, there is usually no movement in the first few time frames when the action starts. The augmented skeleton sequence and the corresponding adjacency matrix are denoted as X and A respectively, and feature extraction can be calculated as:

[0085]

[0086] Here and I are the angular matrix and the identity matrix respectively, and σ(·) is the activation function. W is the learnable matrix. The single-layer spatial graph convolutional network in ST-GCN is used as the skeleton encoder, and the input is the augmented skeleton sequence.

[0087] 3.2.2 Then, truncate c% of the feature-enhanced skeleton sequence X a and discard the remaining sequence to generate a set of dense static anchors (StaticAncor), which can be calculated as:

[0088]

[0089] Here is the data of the t-th frame in the i-th batch of sequences, where t is within our truncated range, i.e., 0 < t < T×c%. N is the batch size, V is the number of nodes, and c is the hyperparameter.

[0090] 3.2.3 Calculate the similarity between the original skeleton S n,1 , and the static anchor S n,2 . To mine the motion area, by comparing the static anchor with the skeleton sequence obtained by normal augmentation, a mask matrix M r is generated, which can be expressed as:

[0091]

[0092] Here X ∈ R C×T×V is the original skeleton data, A ∈ R V×V , n is the number of skeleton sequences, f is the skeleton encoder, T is the data augmentation, c is the hyperparameter, and M r ∈ R C×T×V is a [0,1] mask matrix.

[0093] 3.2.4 Finally, generate a masked skeleton sequence g r , which can be expressed as:

[0094]

[0095] The masked sequence needs to be unified into a fixed-length sequence data, so cubic interpolation is then used to fill the entire skeleton sequence.

[0096] Step 4: Input the action text semantic features obtained in step 2 and the skeleton features obtained in step 3 into the multimodal feature fusion module to achieve the fusion of semantic features and skeleton features; distinguish different action samples by comparing the similarity of feature representations generated by different action samples in the same feature space, and train the encoder through back propagation. Finally, add a fully connected layer with a Softmax activation function to the pre-trained encoder with fixed parameters, and use a fully supervised method to train the classifier to achieve action classification; the specific method is:

[0097] 4.1 First, the data-enhanced feature vector obtained in step 3 is fused with the text feature vector obtained in step 2 to obtain the fused multimodal feature vector, which can be expressed as:

[0098]

[0099] Among them, linear(·) represents linear change, and Alignment fuses skeleton features with text features. and z' are stored in a first-in-first-out memory queue where K represents the queue size, and z' form a positive pair, while z' and the embedding in Q form a negative pair. The InfoNCE loss can be expressed as:

[0100]

[0101] in, Represents the similarity between the embedding of view v and the memory queue Q.

[0102] 4.2 Similarly, mask embedding g r Embed g with the corresponding key value +r is a positive sample pair, where g +r represent and z', while g r and the embedding in Q form a negative sample pair. Therefore, the mask view g r The contrast loss can be expressed as:

[0103]

[0104] Here is the query encoder The parameters of are updated by gradient back propagation, while the key-value encoder The parameters of are updated to the moving average of the query encoder, which can be expressed as:

[0105]

[0106] where m∈[0,1) is a momentum coefficient, which is usually close to 1 to maintain the consistency of embedding in the memory queue.

[0107] Finally, the loss used to optimize the encoder can be formulated as:

[0108]

[0109] Among them, λ is a hyperparameter that balances the two contrast pairs. The overall architecture of the semantic-guided skeleton behavior recognition model is shown in the figure below. Figure 2 shown.

[0110] Through experimental analysis, the model hyperparameters are tuned. Three public large-scale datasets are used, NTU-RGB+D, NTU-RGB+D120, and PKU-MMD. The skeleton data is preprocessed, including original skeleton acquisition, noise removal, and perspective normalization;

[0111] The processed dataset is passed into a multimodal contrastive learning model based on 3D skeleton action representation. In order to mine high-level semantic information in action sequences, it integrates a novel enhancement strategy SkleMask and a multimodal feature fusion method SG-CL. SkeleMask enhances the original skeleton sequence from a temporal perspective and encourages the model to build high-quality challenging difficult contrast pairs, which help the model learn more general and discriminative representations. SG-CL uses LLMs to generate relevant action text descriptions, adding additional semantic information to positive and negative sample pairs, and assisting the model to learn better action representations for action recognition. Therefore, the method of the present invention significantly improves the performance of the model of the present invention on downstream tasks. Extensive experimental results demonstrate the effectiveness of the SG-CLR of the present invention and show that the SG-CLR of the present invention achieves state-of-the-art performance under various evaluation protocols.

[0112] The operating system used in the experiment is Ubuntu 22.04.1, and the deep learning framework used is PyTorch. The specific configurations involved in the experiment are shown in Table 1.

[0113] Table 1 Experimental configuration table

[0114] project Configuration Processor (CPU) 12thGenIntel(R)Core(TM)i9-12900KFcpu@3.20GHz Graphics card (GPU) GeForce RTX3090 operating system Ubuntu 22.04.1 frame PyTorch

[0115] Through sufficient ablation experiments, the effectiveness of the innovative submodules proposed in the present invention is proved, which strengthens the feasibility and performance of the present invention. Through comparative experiments with existing GCN-based methods and Transformer-based methods, the present invention demonstrates performance advantages on three different data sets. This means that the SG-CLR model achieves better results in action classification. Table 2 shows the impact of different action description methods on model performance, Table 3 lists the impact of different action interpolation methods on model performance, and Table 4 shows the results of linear evaluation of the model of the present invention compared to the most advanced methods on multiple action recognition benchmarks.

[0116]

[0117] Table 4: Linear evaluation results of different modalities under different settings on human skeleton datasets NTU-60, NTU-120 and PKU-MMD. J, B, M represent skeleton stream, bone stream and motion stream respectively. J+B+M represents three-stream fusion

[0118]

Claims

1. A self-supervised skeleton behavior recognition method based on action semantics guidance, characterized in that: Firstly, the skeleton video samples to be trained are obtained and preprocessed; secondly, the semantic prototype of each type of action in the preprocessed human skeleton dataset is generated through the action semantic generation module, that is, the corresponding action text semantic features are generated; in addition, the preprocessed human skeleton dataset is input into the feature enhancement module (FAM) to realize skeleton data enhancement and generate skeleton features of the corresponding actions; finally, the multimodal fusion module is used to fuse the action text semantic features and skeleton features of the corresponding actions, so as to assist the model to learn more distinctive and representative action representations from the semantic level and improve the accuracy of fuzzy action recognition. Through the comparative learning method, the self-supervised approach is used to ensure the recognition performance of the model while reducing the dependence on large-scale labeled data, thus realizing skeleton-based human action classification.

2. According to the method of self-supervised skeleton behavior recognition based on action semantics guidance in claim 1, it is characterized in that: The specific steps include: Step 1, collecting skeleton video samples to be trained: using public network resources to obtain a data set containing human skeleton movements, and performing a preprocessing operation, wherein the preprocessing operation includes extracting original skeleton data, removing noise interference and normalizing the view angle to obtain a preprocessed human skeleton data set; Step 2: Generate semantic prototypes for each type of action in the preprocessed human skeleton dataset through the action semantic generation module. Specifically, firstly, design action prompts for the large-scale language model (LLM) to generate behavior descriptions. Then, the generated descriptions are fed to the text encoder to generate the corresponding action text semantic features. Step 3, inputting the human skeleton data set preprocessed in step 1 into the feature enhancement module (FAM) to enhance the skeleton data; the enhanced skeleton data is projected and embedded through the skeleton encoder to generate skeleton features of the corresponding action; In step 4, the action text semantic features obtained in step 2 and the skeleton features obtained in step 3 are input into the multimodal feature fusion module to realize the fusion of semantic features and skeleton features; by setting the loss function, the similarity of the fused feature representations generated by different action samples in the same feature space is compared to distinguish different action samples, and the skeleton encoder is trained by back propagation; finally, a fully connected layer with a Softmax activation function is added after the skeleton encoder trained by back propagation, and a fully supervised method is used to train the classifier to realize action classification.

3. The self-supervised skeleton behavior recognition method based on action semantics guidance according to claim 2 is characterized in that: The step 2 comprises: First, a set of action text descriptions for each type of action in the dataset is generated from the three levels of paragraph, synonym and context reasoning. Then, the semantic prototype representation of the action is obtained through clustering algorithm. Then, the semantic prototype representation is input into the text encoder E t (·), generate the semantic prototype vector: Among them, Text represents the action description generated by LLM, TextEncoder(·) represents the text encoder, Normalization(·) represents the normalization of the aggregated features, and the projection of the generated normalized features is the feature vector that embeds the text information.

4. The method for self-supervised skeleton behavior recognition based on action semantics guidance according to claim 2, characterized in that: The feature enhancement module (FAM) described in step 3 includes a data enhancement (including crop and shear) module and a random skeleton mask data enhancement module; 3.1 In the data enhancement module, given the original skeleton sequence S, the enhanced skeleton sequence is obtained by cropping and shearing transformation, and the query encoder is used and momentum updated key-value encoder Extract the spatiotemporal features of the human skeleton, and then use global average pooling (GAP) to obtain the query embedding z and key value embedding 3.2 Through the random skeleton mask data enhancement module, action recognition is performed on partial sequences of temporal action intervals: 3.2.1 Visualize the original skeleton sequence. Since the action usually occurs in a very small local temporal region, there is no movement in the first few time frames of the action. The enhanced skeleton sequence in the data enhancement module and the adjacency matrix representing the physical connection of the human body joints are represented as X and A respectively. The feature extraction calculation is: here, is the angle matrix, I is the unit matrix, σ(·) is the activation function; W is the learnable matrix; a skeleton encoder is used to input the enhanced skeleton sequence; 3.2.2 Feature X of the enhanced skeleton sequence a Perform a truncation c%, discard the remaining sequences, and generate a set of dense static anchors (StaticAncor): Here, is the data of the t-th frame in the i-th batch of sequences, where t is within the truncated range, i.e., 0 < t < T×c%, N is the batch size, V is the number of nodes, and c is a hyperparameter; 3.2.3 Calculate the original skeleton S n,1 With static anchor point S n,2 In order to mine the motion area, the static anchor points are compared with the skeleton sequence obtained in the data enhancement module to generate a mask matrix M. r : Here, X∈R C×T×V is the original skeleton data, A∈R V×V , n is the number of skeleton sequences, f is the skeleton encoder, T is the data augmentation, c is the hyperparameter, M r ∈R C×T×V is a [0,1] mask matrix; 3.2.4 Mask some time frame data according to the mask matrix to generate a masked skeleton sequence gr, expressed as: The masked sequence needs to be unified into a fixed-length sequence data, and multiple interpolations are used to adjust the masked skeleton sequence to the target length.

5. The method for self-supervised skeleton behavior recognition based on action semantics guidance according to claim 2, characterized in that: The specific method of step 4 is: 4.1 First, the data-enhanced feature vector obtained in step 3.1 is fused with the text feature vector obtained in step 2 to obtain the corresponding fused multimodal feature vector: in, z′ represents the multimodal feature vector fused with the feature vector after data enhancement by cropping and shearing transformation, linear(·) represents linear change, and Alignment fuses the skeleton feature with the text feature; define a first-in-first-out memory queue The embedding m in Q i is the key value embedding z' of the previous round, where K represents the queue size, and z' form a positive sample pair, while z' and the embedding m in Q i Form negative sample pairs; InfoNCE loss can be expressed as: in, represents the similarity between the embedding z' and the memory queue Q; 4.2 Similarly, the masked skeleton sequence g r The query embedding is obtained through global average pooling (GAP) and and z′ respectively constitute two positive sample pairs, and we use represent and z′, and and the embedding in Q form a negative sample pair; therefore, the mask view The contrast loss is expressed as: here, To query the encoder, its parameters are updated via gradient backpropagation, is a key-value encoder, and The parameters of the query encoder are used The moving average update is expressed as: i k =mθ k +(1-m)θ q , (9) Where m∈[0,1) is a momentum coefficient, which is usually close to 1 to maintain the consistency of embedding in the memory queue; Finally, the encoder is optimized with loss L: Among them, λ is a hyperparameter that balances the two contrast pairs; Update the encoder according to the calculated loss function L through the gradient descent algorithm Parameters, iterative training is performed until the model reaches the set number of training rounds, and the encoder is retained As a feature extractor, it is used in subsequent downstream tasks to achieve action classification.

6. A self-supervised skeleton behavior recognition system based on action semantics guidance, characterized in that: include: The action semantic generation module is used in step 2. It uses a clustering algorithm to implement a set of semantic prototype representations of action text descriptions at three levels: paragraph, synonym, and context reasoning for each type of action. Then, the action description is generated by carefully designing action prompts for LLMs. The generated description is fed into the text encoder to generate the corresponding action text semantic features. Feature Enhancement Module (FAM), used in step 3, firstly, implements conventional enhancement of skeleton sequence through cropping and shearing transformation; secondly, sets static anchor points, obtains mask matrix by comparing with skeleton sequence of conventional data enhancement, implements random skeleton mask data enhancement according to mask matrix, and realizes mining of motion area; The action semantics-guided multimodal fusion module is used in step 4 to achieve skeleton and text feature fusion by superimposing the action semantic prototype vector on the skeleton vector, assisting the model to learn more distinctive and representative action representations from the semantic level, and improving the accuracy of fuzzy action recognition.

7. A self-supervised skeleton behavior recognition device based on action semantics guidance, characterized in that: include: Memory for storing computer programs; A processor is used to implement the self-supervised skeleton behavior recognition method based on action semantics guidance described in steps 1 to 4 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is possible to implement self-supervised skeleton behavior recognition based on action semantics guidance based on the method described in steps 1 to 4.

Images