Skeleton action recognition method and system based on mixed skeleton local and global comparative learning

Abstract

The invention discloses a skeleton action recognition method and system based on mixed skeleton local and global comparative learning, and relates to artificial intelligence and human body action recognition technologies, and the method comprises the steps: obtaining a to-be-trained skeleton action sample which comprises a plurality of skeleton action sequences; performing basic data enhancement and mixed skeleton enhancement on the skeleton action sequence to obtain an enhanced skeleton action sequence; for the enhanced skeleton action sequence, adopting a multi-head attention mask module to learn soft attention mask features, expanding the soft attention mask features obtained based on learning into global features, and generating a contrast; training a skeleton action recognition model by using the generated comparison pair; and inputting the to-be-recognized sequence into the skeleton action recognition model obtained by training to output a skeleton action recognition result. According to the invention, a basic skeleton data enhancement strategy and a mixed skeleton data enhancement strategy are provided, and robustness and accuracy of skeleton human motion recognition are improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and human motion recognition technology, and in particular to a skeleton motion recognition method and system based on hybrid skeleton local-global comparison learning. Background Technology

[0002] Human action recognition is a crucial research topic in computer vision, aiming to understand human behavior and distinguish human actions. With the rapid development of artificial intelligence and computer vision technologies, intelligent interaction between human actions and computers has gradually become a popular choice in many industries. Under this trend, human action recognition technology has naturally become a core technology for intelligent interaction. With the booming development of deep learning technology, action recognition based on fully supervised skeletons has made significant progress. However, these methods have certain limitations in application because they require large-scale labeled data, which is not always readily available.

[0003] In contrast, unsupervised skeleton action recognition has attracted widespread attention. By utilizing unlabeled data, these methods can learn more general features, thus helping to alleviate overfitting and reducing reliance on large amounts of labeled training data. Despite some progress made by current unsupervised methods, several unresolved issues remain. The semantic distinction of human action features is often represented by local body parts (such as legs or hands), which aids skeleton-based action recognition. However, skeleton sequences belonging to the same action category may exhibit significant differences in feature space. Existing contrastive learning-based skeleton action recognition methods neglect the consideration of local feature similarity in skeleton sequences, thus limiting the accuracy of the model's representation of local actions. Summary of the Invention

[0004] This application provides a skeleton action recognition method and system based on hybrid skeleton local-global contrastive learning. It proposes multiple skeleton data augmentation strategies, namely basic skeleton data augmentation strategy and hybrid skeleton data augmentation strategy, to generate different motion views for skeleton sequences and improve the robustness and accuracy of skeleton human motion recognition.

[0005] This application provides a skeleton action recognition method based on hybrid skeleton local-global contrastive learning, including: Obtain skeleton action samples to be trained, wherein the skeleton action samples contain multiple skeleton action sequences; The skeleton motion sequence is subjected to basic data augmentation and hybrid skeleton augmentation respectively to obtain an enhanced skeleton motion sequence; For the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features, and the learned soft attention mask features are expanded into global features to generate comparison pairs; The skeleton action recognition model is trained using the generated comparison pairs; The skeleton sequence to be identified is input into the skeleton action recognition model obtained through training, so as to output the skeleton action recognition result.

[0006] Optionally, obtaining the skeleton action samples to be trained includes: Collect publicly available human skeleton motion data and preprocess the collected human skeleton motion data. Performing basic data augmentation and hybrid skeleton augmentation on the skeleton motion sequence includes: Spatial transformation (Shear) and temporal transformation (Crop) are applied to the skeleton motion sequence to obtain the basic enhanced skeleton sequence; and, The skeletal motion sequence is cropped and combined. In the cropping, two discrete uniform distributions are used to determine the number of body parts to be cropped and the duration of the cropped skeletal fragments. In the combination, the cropped skeletal fragments are randomly combined to generate a hybrid enhanced skeletal sequence.

[0007] Optionally, for the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features. These learned soft attention mask features are then expanded into global features to generate comparison pairs, including: The basic augmented skeleton sequence is randomly converted into different data augmentation versions x. q and x k ; Using the corresponding encoder E q and E k Different data enhancement versions x q and x k Embedded into feature space , , where f q , f k ∈R n×C f E k According to E q Update parameters using momentum update mechanism M∈[0,1) is the momentum coefficient; embedding feature space and Perform global average pooling and input them separately. and The embedded features after global average pooling are projected onto the same latent space to obtain the query embedding z. q and key-value embedding z k , where z q , z k ∈R Cz ; In each iteration, query the embedding z. q With key-value embedding z k Form negative sample pairs.

[0008] Optionally, for the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features. These learned soft attention mask features are then expanded into global features to generate comparison pairs. This process also includes: In global comparative learning, InfoNCE is used to perform global instance identification of skeleton actions.

[0009] Optionally, for the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features. These learned soft attention mask features are then expanded into global features to generate comparison pairs. This process also includes: Based on data-enhanced version x q Multi-head self-attention mask (MHSAM) is used to segment the feature embeddings of local contrastive learning, and contrast pairs are constructed between the segmented local embeddings.

[0010] Optionally, training the skeleton action recognition model using the generated contrast pairs includes: The loss functions for salient and insignificant features satisfy: in, Indicates positive sample embedding. This indicates the positive sample embedding after clipping and mixing. This indicates the negative sample embedding. This represents the negative sample embedding after clipping and mixing. Indicates the temperature coefficient; The total loss function for local contrastive learning is defined to satisfy: Where u is the weighting coefficient.

[0011] Optionally, training the skeleton action recognition model using the generated contrast pairs includes: The total loss function satisfies: in, Indicates global loss. Represents the global distance between samples. This indicates local contrast loss.

[0012] This application also proposes a skeleton action recognition system based on hybrid skeleton local-global contrastive learning, including a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the skeleton action recognition method based on hybrid skeleton local-global contrastive learning as described above.

[0013] The method in this application proposes a variety of skeleton data augmentation strategies, namely basic skeleton data augmentation strategy and hybrid skeleton data augmentation strategy, to generate different motion views for skeleton sequences and improve the robustness and accuracy of skeleton human motion recognition.

[0014] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the basic process of the skeleton motion recognition method according to an embodiment of this application; Figure 2 This is a schematic diagram of the hybrid skeleton strategy of the skeleton action recognition method in this application embodiment; Figure 3 This is a schematic diagram of the skeleton action recognition model architecture of the skeleton action recognition method in this application embodiment; Figure 4 This is a schematic diagram of the MHSAM method for skeleton motion recognition in an embodiment of this application; Figure 5 This is a diagram illustrating normal dot product operation; Figure 6 This is a schematic diagram of the dot product operation using a soft attention mask in an embodiment of this application. Detailed Implementation

[0016] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0017] The method in this application first acquires skeleton video samples to be trained; secondly, it performs basic data augmentation and hybrid skeleton augmentation on the skeleton sequences respectively; then, it uses a multi-head attention masking module to learn soft attention mask features from the hybrid skeleton sequences, suppressing insignificant local features and highlighting salient local features, thereby making similar local features closer in the feature space. Through the attention masking mechanism, local features can be learned under different data augmentation views; furthermore, by expanding salient and insignificant features into global features, a large number of contrast pairs are generated, thereby guiding the network to learn the semantic representation of the entire skeleton, realizing the action recognition task based on skeleton data. Specifically, this application provides a skeleton action recognition method based on hybrid skeleton local-global contrastive learning, such as... Figure 1 As shown, it includes the following steps: In step S101, a skeleton motion sample to be trained is obtained, which includes multiple skeleton motion sequences. In some embodiments, obtaining the skeleton motion sample to be trained includes: collecting publicly available human skeleton motion data and preprocessing the collected human skeleton motion data. For example, in a specific example, publicly available human skeleton motion datasets can be collected online; here, three publicly available large-scale datasets are used: NTU-RGB+D, NTU-RGB+D120, and PKU-MMD. Preprocessing operations on the skeleton data include raw skeleton acquisition, noise removal, and viewpoint normalization. Specifically, skeleton data preprocessing may include: Raw skeleton acquisition refers to data captured from different sensors or cameras, typically in various formats such as joint coordinates, depth images, or RGB images. During the preprocessing stage, this raw data needs to be extracted and normalized for subsequent analysis, including calibrating and synchronizing data from different sensors to ensure they are available in the same temporal and spatial coordinate system.

[0018] Noise removal is a critical step because raw data often contains noise from sources such as sensor errors, environmental interference, or motion blur. This noise can affect the accuracy of subsequent analysis and applications. Therefore, data preprocessing typically includes filtering and smoothing operations to remove this noise, thereby improving the reliability of the skeleton data.

[0019] Viewpoint normalization is used to ensure consistency of skeleton data across different viewpoints and camera settings. This typically involves mapping the skeleton data to a standard coordinate system so that data from different scenes can be more easily compared and analyzed.

[0020] Preprocessing skeleton data is a crucial step that helps ensure data quality and consistency, thereby improving the performance and accuracy of computer vision and human motion analysis applications.

[0021] In step S102, the skeleton motion sequence is subjected to basic data augmentation and hybrid skeleton augmentation respectively to obtain an enhanced skeleton motion sequence.

[0022] In some embodiments, performing basic data augmentation and hybrid skeleton augmentation on the skeleton action sequence includes: applying spatial transformation Shear and temporal transformation Crop to the skeleton action sequence to obtain a basic augmented skeleton sequence; and, The skeletal motion sequence is cropped and combined. In the cropping, two discrete uniform distributions are used to determine the number of body parts to be cropped and the duration of the cropped skeletal fragments. In the combination, the cropped skeletal fragments are randomly combined to generate a hybrid enhanced skeletal sequence.

[0023] In a specific example, the original skeleton action sequence is subjected to a basic data augmentation strategy and a hybrid skeleton augmentation strategy to obtain a basic augmented skeleton sequence and a hybrid skeleton sequence, respectively.

[0024] In some specific examples, basic data augmentation includes spatial transformations (Shear) and temporal transformations (Crop) on skeleton action sequences. Hybrid data augmentation includes pruning and combining steps. Specifically, such as... Figure 2 As shown, in the trimming step, based on the topological information of the skeleton data, the skeleton joints are divided into 5 subsets P = {left hand, right hand, left leg, right leg, torso}. Next, two discrete uniform distributions Bs~U(Bsl, Bsu) and Bt~U(Btl, Btu) are used to determine the number of body parts to be trimmed and the duration of the trimmed skeleton fragments. Samples are taken once in each training iteration to obtain Ns and Nt. Then, n body parts are randomly selected from P to obtain the trimmed skeleton joints S, and the starting frame ts is randomly sampled within a valid range that ensures the integrity of the trimmed skeleton fragments. Combining Nt and ts, the corresponding temporal trimming region T is obtained. Finally, a spatiotemporal trimming operation is performed on x to obtain the trimmed skeleton fragments. In the combination step, the trimmed skeleton fragments (named the trimmed view) and the remaining skeleton motion sequence (called the truncated view) are randomly combined to generate a hybrid skeleton sequence. Therefore, the hybrid skeleton sequence consists of the trimmed view and the truncated view.

[0025] In step S103, for the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features, and the learned soft attention mask features are expanded into global features to generate comparison pairs.

[0026] like Figure 3As shown, specifically, based on the basic enhanced skeleton sequence and the hybrid skeleton enhanced sequence obtained above, a large number of contrast pairs are generated for the basic enhanced skeleton sequence and the hybrid skeleton enhanced sequence through the global contrast learning module and the local contrast learning module, respectively, thereby guiding the network to learn the semantic representation of the entire skeleton and realize the skeleton action recognition task.

[0027] In step S104, the skeleton action recognition model is trained using the generated comparison pairs; In step S105, the skeleton sequence to be identified is input into the trained skeleton action recognition model to output the skeleton action recognition result.

[0028] The method in this application proposes a variety of skeleton data augmentation strategies, namely basic skeleton data augmentation strategy and hybrid skeleton data augmentation strategy, to generate different motion views for skeleton sequences and improve the robustness and accuracy of skeleton human motion recognition.

[0029] In some embodiments, for the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features, and the learned soft attention mask features are expanded into global features to generate comparison pairs, including: The basic augmented skeleton sequence is randomly converted into different data augmentation versions x. q and x k .like Figure 3 As shown, in a specific example, for the input data of global contrastive learning, Shear and Crop are used as augmentation strategies. In this embodiment, this data augmentation combination is referred to as basic data augmentation N. N randomly transforms the original skeleton sequence X into two different data augmentation versions x. q and x k As the opposite.

[0030] Using the corresponding encoder E q and E k Different data enhancement versions x q and x k Embedded into feature space , , where f q , f k ∈R n×C f E k According to E q Update parameters using momentum update mechanism M∈[0,1) is the momentum coefficient that satisfies: .

[0031] Then, the embedded feature space and Global average pooling is performed, and the results are then fed into the predictor. and The embedded features after global average pooling are projected onto the same latent space to obtain the query embedding z. q and key-value embedding z k， Where z q , z k ∈R Cz .

[0032] In some examples, in each iteration, the prior embedding z is used as the basis. k , with the newly calculated z q Form negative pairs. Specifically, a queue can be used. To store the previous embedded z k In each iteration, M k This provides a large number of negative pairs for contrastive learning. In each iteration, z k It was previously stored in M k M was calculated from the samples in the dataset. k Remove from the list according to the storage order to obtain , with the newly calculated z q This results in a large number of negative pairs.

[0033] In some embodiments, for the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features, and the generated comparison pairs are expanded based on the learned soft attention mask features into global features. In global comparative learning, InfoNCE is used to perform global instance discrimination of skeleton actions, as shown in the following formula. Among them, z q z k and m i All parameters are standardized. τ is a hyperparameter (set to 0.2 in the experiment). This represents z and m embedded from the memory bank. i The similarity between samples. In this loss function, the global distance between samples is calculated by the dot product.

[0034] In some embodiments, for the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features, and the generated comparison pairs are expanded based on the learned soft attention mask features into global features. Based on data-enhanced version x q Multi-head self-attention mask (MHSAM) is used to segment the feature embeddings of local contrastive learning, and contrast pairs are constructed between the segmented local embeddings.

[0035] In a specific example, a local contrastive learning module is used to generate different positive and negative sample pairs for the skeleton action sequence from the perspective of local features. These include salient feature sample pairs and non-salient feature sample pairs, thereby generating different motion views for skeleton learning. This enables the model to better learn the local and global features of the skeleton action sequence and improve the accuracy of skeleton action recognition.

[0036] In x q Based on this, additional data augmentation is performed, and then multi-head attention masking (MHSAM) is used to segment the feature embeddings of local contrastive learning. Finally, rich contrast pairs are constructed between the segmented local embeddings to pursue better local feature mining.

[0037] The input to the MHSAM module is a tensor of size b×n×c, such as Figure 4 As shown, b represents the batch size, n represents the length of a single data item, and c represents the number of data channels.

[0038] MHSAM is designed to embed the encoder output at the feature level. This is because the encoder in local comparison learning differs from the encoder in global comparison learning. q Shared parameters, therefore... f mix = E q (x mix ;θ q ),in f mix ∈R n×Cf .

[0039] The formulas for calculating Q, K, and V in the multi-head self-attention mechanism are as follows: Among them, W q W k W v Let be the parameter matrix of the linear network. Then, the attention features are described using the softmax function, as shown in the following formula: in This is the normalization scaling factor. After calculation, x... attn The process is then transformed into a simple linear network projection for adjustment. Finally, a sigmoid function is used to generate a soft mask Ms, as shown in the following formula: Here, λ is a hyperparameter that adjusts the tolerance of neutral features. The larger the value of λ, the more easily the mask value becomes polarized.

[0040] After obtaining the mask, the feature-level embeddings extracted by the encoder are separated into salient and non-salient features using the mask.

[0041] During mask pooling, the input f mix and f k We obtain f by transforming it in significant and non-significant ways respectively. s f ns f ks and f kns The formula for mask pooling is shown below: Where I is the identity matrix and n is the length of the data in the dimension to be pooled.

[0042] In some embodiments, training the skeleton action recognition model using the generated contrast pairs includes: establishing loss functions for salient and non-salient features.

[0043] After obtaining the embedding from the mask pool, we get q. s =P q (f s ), q ns =P q (f ns ), k s =P k (f ks ) and k ns = P k (f kns In local contrastive learning, q s and k s Set it to face, and set q ns and k ns Set it to face, and set q s and q ns Let's assume it's a negative pair. Then, the loss function expansion for salient and insignificant features is as shown in the following formula: in, Indicates positive sample embedding. This indicates the positive sample embedding after clipping and mixing. This indicates the negative sample embedding. This represents the negative sample embedding after clipping and mixing. This represents the temperature coefficient.

[0044] In L s and L nsIn the estimation, attention is used to calculate local similarity. Let the normal dot product operation be as follows: Figure 5 As shown. The mask generated by MHSAM highlights the parts of interest at the feature level, causing the dot product operation to tend to emphasize local similarities in the image, such as... Figure 6 As shown. q s With q ns The dot product represents the opposition between them. In local contrastive learning, a memory M storing global feature embeddings is used. k To provide negative sample pairs, it is worth noting that the local-to-global dot product, like the local-to-local product, emphasizes local similarity.

[0045] Finally, the total loss function for local contrastive learning is defined to satisfy: Where u is the weighting coefficient.

[0046] In some embodiments, training the skeleton action recognition model using the generated comparison pairs includes: The total loss function satisfies: in, Indicates global loss. Represents the global distance between samples. This indicates local contrast loss.

[0047] Experimental analysis was conducted to optimize the model hyperparameters. Three publicly available large-scale datasets, NTU-RGB+D, NTU-RGB+D120, and PKU-MMD, were used in the application examples of this application. Preprocessing operations were performed on the skeleton data, including raw skeleton acquisition, noise removal, and viewpoint normalization. The processed dataset is fed into the skeleton action recognition model. First, the skeleton sequence undergoes basic data augmentation and hybrid skeleton augmentation strategies to generate basic augmented skeleton sequences and hybrid augmented skeleton sequences. These sequences are then input into the global and local contrastive learning modules, respectively, to generate a large number of contrast pairs, thereby guiding the network to learn the semantic representation of the entire skeleton. Finally, based on the learning objective of addressing the information bottleneck, this application improves the training loss to learn information-rich yet compact latent representations in the skeleton data, guiding the model to understand the maximum information representation.

[0048] The operating system used in the experiment was Ubuntu 22.04.3, and the deep learning framework used was PyTorch. The specific configurations involved in the experiment are shown in Table 1.

[0049] Table 1 Experimental Configuration Table project Configuration CPU 13th Gen Intel(R) Core(TM) i9-13900KF GPU GeForce RTX 4080 operating system Ubuntu 22.04.3 frame PyTorch Extensive ablation experiments demonstrate the effectiveness of the innovative sub-modules proposed in this application, enhancing its feasibility and performance. Comparative experiments with existing contrastive learning-based skeleton action recognition methods show performance advantages on three different datasets, indicating that the model achieves better results in action recognition.

[0050] The method in this application addresses the problem that existing contrastive learning-based skeleton action recognition methods neglect the consideration of the local feature similarity of skeleton sequences, thus limiting the accuracy of the model's local representation of actions. This application introduces a local-global contrastive learning network based on hybrid skeleton sequences to solve this problem, allowing the model to better learn the local features of skeleton actions and improving the model's accuracy in recognizing actions.

[0051] This application includes basic data augmentation strategies and a global contrastive learning module. These modules generate different positive sample pairs for the skeleton sequence from a global feature perspective, which helps the system capture global feature information of skeleton actions, thereby improving the accuracy of action recognition.

[0052] This application introduces a hybrid skeleton strategy and a local contrastive learning module to generate different positive and negative sample pairs for the skeleton sequence from the perspective of local features, including salient feature sample pairs and non-salient feature sample pairs, thereby generating different motion views for the skeleton sequence. This enables the model to better learn the local and global features of the skeleton action sequence and improve the accuracy of skeleton action recognition.

[0053] This application also proposes a skeleton action recognition system based on hybrid skeleton local-global contrastive learning, including a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the skeleton action recognition method based on hybrid skeleton local-global contrastive learning as described above.

[0054] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0055] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0056] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0057] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A skeleton action recognition method based on hybrid skeleton local-global contrastive learning, characterized in that, include: Obtain skeleton action samples to be trained, wherein the skeleton action samples contain multiple skeleton action sequences; The skeleton motion sequence is subjected to basic data augmentation and hybrid skeleton augmentation respectively to obtain an enhanced skeleton motion sequence; For the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features, and the learned soft attention mask features are expanded into global features to generate comparison pairs; The skeleton action recognition model is trained using the generated comparison pairs; The skeleton sequence to be identified is input into the skeleton action recognition model obtained through training, so as to output the skeleton action recognition result.

2. The skeleton action recognition method based on hybrid skeleton local-global contrastive learning as described in claim 1, characterized in that, Obtaining training skeleton motion samples includes: Collect publicly available human skeleton motion data and preprocess the collected human skeleton motion data. Performing basic data augmentation and hybrid skeleton augmentation on the skeleton motion sequence includes: Spatial transformation (Shear) and temporal transformation (Crop) are applied to the skeleton motion sequence to obtain the basic enhanced skeleton sequence; and, The skeletal motion sequence is cropped and combined. In the cropping, two discrete uniform distributions are used to determine the number of body parts to be cropped and the duration of the cropped skeletal fragments. In the combination, the cropped skeletal fragments are randomly combined to generate a hybrid enhanced skeletal sequence.

3. The skeleton action recognition method based on hybrid skeleton local-global contrastive learning as described in claim 2, characterized in that, For the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features. These learned soft attention mask features are then expanded into global features to generate comparison pairs, including: The basic augmented skeleton sequence is randomly converted into different data augmentation versions x. q and x k ; Using the corresponding encoder E q and E k Different data enhancement versions x q and x k Embedded into feature space , , where f q , f k ∈R n×C f , of which E k According to E q Update parameters using momentum update mechanism M∈[0,1) is the momentum coefficient; embedding feature space and Perform global average pooling and input them separately. and The embedded features after global average pooling are projected onto the same latent space to obtain the query embedding z. q and key-value embedding z k , where z q , z k ∈R Cz ; In each iteration, query the embedding z. q With key-value embedding z k Form negative sample pairs.

4. The skeleton action recognition method based on hybrid skeleton local-global contrastive learning as described in claim 3, characterized in that, For the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features. These learned soft attention mask features are then expanded into global features to generate comparison pairs. In global comparative learning, InfoNCE is used to perform global instance identification of skeleton actions.

5. The skeleton action recognition method based on hybrid skeleton local-global contrastive learning as described in claim 3, characterized in that, For the enhanced skeleton action sequence, a multi-head attention mask module is used to learn soft attention mask features. These learned soft attention mask features are then expanded into global features to generate comparison pairs. Based on data-enhanced version x q Multi-head self-attention mask (MHSAM) is used to segment the feature embeddings of local contrastive learning, and contrast pairs are constructed between the segmented local embeddings.

6. The skeleton action recognition method based on hybrid skeleton local-global contrastive learning as described in claim 5, characterized in that, Training the skeleton action recognition model using the generated contrast pairs includes: The loss functions for salient and insignificant features satisfy: in, Indicates positive sample embedding. This indicates the positive sample embedding after clipping and mixing. This indicates negative sample embedding. This represents the negative sample embedding after clipping and mixing. Indicates the temperature coefficient; The total loss function for local contrastive learning is defined to satisfy: Where u is the weighting coefficient.

7. The skeleton action recognition method based on hybrid skeleton local-global contrastive learning as described in claim 6, characterized in that, Training the skeleton action recognition model using the generated contrast pairs includes: The total loss function satisfies: in, Indicates global loss. Represents the global distance between samples. This indicates local contrast loss.

8. A skeleton action recognition system based on hybrid skeleton local-global contrastive learning, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the steps of the skeleton action recognition method based on hybrid skeleton local-global contrastive learning as described in any one of claims 1 to 7.

Images