Zero-sample skeleton action recognition method and system with self-adaptive test time

Abstract

The invention discloses a test time adaptive zero sample skeleton action recognition method and system, which uses text embedding to guide skeleton embedding to generate a more different same semantic view, and at the same time, Gaussian noise and a mask strategy are used to generate a richer enhanced view, so as to improve the recognition accuracy of the zero sample skeleton action. The deficiency of object auxiliary information by a skeleton sequence is made up to a certain extent; a self-adaptive prompt is generated through the combination of text embedding and skeleton embedding, and the semantic space difference between text and skeleton modals is bridged; in the test time self-adaption process, prediction with higher confidence is considered preferentially, prediction with low confidence is eliminated, and the problem of serious distribution offset can be effectively relieved.

Description

Technical Field

[0001] The present invention relates to a language-guided test-time adaptive zero-sample skeletal action recognition method and system, a test-time adaptive method for enhancing zero-sample generalization capability, and belongs to the field of computer vision technology. Background Art

[0002] Skeletal point data is lightweight, compact, and robust to foreground occlusion, background interference, and viewpoint changes. Consequently, skeletal action recognition has attracted increasing attention and has found widespread application in fields such as intelligent surveillance, virtual reality, and human-computer interaction. Advances in sensor acquisition equipment and developments in pose estimation algorithms have led to the continuous development of large-scale datasets, spurring research on robust baseline models. However, in real-world scenarios, unknown categories are inevitably encountered. For example, a model trained on a large dataset of everyday human behavior demonstrates excellent action recognition capabilities, but when tested on samples of the action category "waving a knife," the model's performance is disappointing because the category was not observed during training. Improving the model's zero-shot generalization capability to ensure reliable real-world application is currently a top priority.

[0003] Missing object information is a major challenge that existing zero-shot skeletal action recognition technologies must address. Human skeleton data has a natural graph structure, so when combined with an adjacency matrix, only around 20 joint points are sufficient to describe several human actions. However, the skeletal representations of many actions are extremely similar. For example, "combing hair" and "wiping face" both have the semantics of "moving the hand toward the head and briefly pausing" when aided by missing object information. This means that the model's generalization to these two categories is ambiguous, and can even cause known distributions to become unknown, exacerbating distribution shift.

[0004] Existing techniques focus on aligning features between the skeletal and text modalities to enhance zero-shot generalization capabilities in a multimodal space. However, this work ignores the inherent flaw of skeletal point data in lacking object information, and the more serious offset problem can also lead to a decline in model performance. Summary of the Invention

[0005] Purpose of the invention: In response to the problems and shortcomings in the prior art, the present invention provides a language-guided test-time adaptive zero-sample skeletal action recognition method and system to make up for the inherent defect of missing object information in skeletal point data and alleviate the situation of severe distribution deviation, thereby improving the zero-sample generalization ability of the model and achieving more accurate action recognition.

[0006] Technical solution: A language-guided test-time adaptive zero-shot skeletal action recognition method, comprising the following steps:

[0007] Step 1: Randomly sample a certain number of frames from the skeleton sequence of the invisible category in the test phase as test samples; the skeleton sequence of the invisible category is the zero sample.

[0008] Step 2: Load the target baseline model as a feature extractor / mapper and freeze the parameters.

[0009] Step 3: Extract the skeleton features and text features of the current test sample.

[0010] Step 4: The text features are used as auxiliary guiding information and fused with the skeleton features to obtain prompt words that can be adaptively distributed during testing.

[0011] Step 5: Mix the randomly generated Gaussian noise, the randomly generated mask matrix, and the randomly cropped partial text features of known categories with the skeleton features to obtain several skeleton-enhanced view sequences with the same semantics.

[0012] Step 6: Perform stability mapping between the cue word obtained in Step 4 and the skeleton-enhanced view sequence obtained in Step 5. Simultaneously, calculate the similarity between the text features of the cue word and the skeleton features of the action sequence as a classification basis, and obtain the probability distribution of each unseen category prediction under all enhanced views.

[0013] Step 7: The probability distribution output of each bone-enhanced view on the unseen category is weighted from low to high according to the size of the self-entropy value to obtain the weighted entropy. Predictions (probability distributions) with too high self-entropy values (exceeding the set threshold) are ignored, and the weights of the retained views sum to 1.

[0014] Step 8: Optimize the adaptively distributed prompt words based on minimizing the weighted entropy obtained in step 7;

[0015] Step 9: Output the current classification result.

[0016] Given a set of human motion skeleton sequences This includes the visible subset and the invisible subset and They are the corresponding visible subsets and the invisible subset The goal of zero-shot skeletal action recognition is to use only the visible subset With the corresponding visible label Train and predict the input sequence Category The present invention does not require complex pre-training, but directly realizes the recognition of unseen categories in the testing phase based on fine-tunable prompt words.

[0017] The process of extracting skeleton features and text features in step 3 is:

[0018]

[0019] in, and They represent the frozen skeleton feature extractor and text feature extractor respectively, x is the current input sample, superscripts s and t represent the skeleton mode and text mode respectively, f s is the bone feature, f t is a text feature.

[0020] The prompt word pt in step 4 is anchored by text features as guidance information, and the skeleton features of the sample are used as displacement in the feature space. get:

[0021]

[0022] In step 4, the prompt words are optimized. The optimization goals are:

[0023]

[0024] in, represent and pt is a hint word that is expected to be optimized and can adapt to the distribution during testing.

[0025] The generation method of the skeleton enhanced view sequence Aug in step 5 is as follows:

[0026]

[0027] Among them, Aug noise is the enhanced view obtained by mixing random Gaussian noise, Aug mask is the enhanced view obtained by applying the random mask strategy, It is an enhanced view of language guidance obtained by randomly cropping the text features of the seen category c and mixing them with the current test skeleton sequence. ⊙ represents the operation of the Hadamard product, M is a randomly generated 0 / 1 matrix. ∈ is a normal distribution with mean μ and variance σ. α c It is a random number (0,1) that determines the text feature corresponding to category c For the current sample bone feature f s After obtaining the skeleton enhancement view sequence Aug, the present invention updates the optimization target of step 4 to:

[0028]

[0029] Among them, Aug jis the jth of all K skeleton-enhanced view sequences, the base of log is 2, is the probability of the jth bone-enhanced view and the prompt word being the i-th category. More specifically, the sample Belong to category The probability calculation formula is:

[0030]

[0031] in, is the frozen stability mapper, sim(·) represents the computation of cosine similarity, and τ is the hyperparameter temperature coefficient.

[0032] In the method of the present invention, based on the similarity, the probability distribution p of each bone enhancement view predicted on the unseen category can be obtained. i The way to calculate the weighted entropy in step 7 is:

[0033]

[0034] Among them, H represents the probability distribution p i Calculate the self-entropy. Represents an indicator. Prediction results with confidence lower than θ (entropy value is too high) will be ignored. The prediction results corresponding to the retained i-th view will also be weighted according to the confidence

[0035] A language-guided test-time adaptive zero-shot skeletal action recognition system is presented, comprising a frozen feature extractor / mapper, a language-guided multi-features augmentation (LG-MFA) module, a language-guided prompt generator (LGPG), and a confidence-weighted selection (CWS) module. To highlight the effectiveness of the proposed test-time approach, during the test phase, the frozen feature extractor / mapper refers to the encoders in the CLIP, CTR-GCN, and variational autoencoder architectures, serving as the text feature extractor, skeletal feature extractor, and noise-removing mapper, respectively. The language-guided multi-feature generation module is used to fuse text features as auxiliary guiding information with skeleton features to obtain prompt words that can be adaptively distributed during testing; text features are used to guide skeleton features to generate more differentiated semantic views through feature fusion, which to a certain extent makes up for the lack of auxiliary information of the object in the skeleton sequence; at the same time, randomly generated Gaussian noise, randomly generated mask matrix and randomly cropped part of the text features of known categories are mixed with the skeleton features to obtain several semantically enhanced skeleton view sequences; Gaussian noise and mask strategies are used to generate richer enhanced views Augmented Reality noise and Aug mask The language-guided prompt generation module regards text features as the starting point for generating prompt words in a high-dimensional feature space and skeleton features as the required displacement. The two work together to obtain fine-tunable adaptive prompt words, thereby bridging the semantic space gap between text and skeleton modalities. The confidence-weighted selection module judges the confidence of the prediction results of a view based on the self-entropy of the prediction results under different views during the process of fine-tuning the prompt words during testing. By giving higher weights to give priority to predictions with higher confidence (lower self-entropy values) and excluding predictions with low confidence, the problem of severe distribution shift can be effectively alleviated.

[0036] A computer-readable storage medium stores a computer program for executing the above-mentioned language-guided test-time adaptive zero-sample skeletal action recognition method.

[0037] Beneficial effects: Compared with the existing technology, the language-guided test time adaptive zero-sample skeletal action recognition method and system of the present invention have achieved the most advanced results in a total of 7 divisions of three data, namely NTU-RGB+D 60, NTU-120+D 120 and PKU-MMD. The present invention realizes differentiated processing of semantically similar categories through the language-guided multi-feature generation module (LG-MFA) and the language-guided prompt word generation module (LGPG), making up for the inherent defect of missing object information. In addition, the method realizes the paradigm of test time adaptation and optimizes the test process through the confidence-weighted selection strategy (CWS), ensuring that the situation of severe distribution deviation is alleviated. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0039] Figure 2 This is a framework diagram of the language-guided test-time adaptive zero-shot skeletal action recognition system of the present invention;

[0040] Figure 3 Schematic diagram of the confusion matrix comparison of the prediction results of the existing method and the present invention under the 10 unseen class divisions of the NTU-120 dataset. DETAILED DESCRIPTION

[0041] The present invention is further illustrated below with reference to specific examples. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0042] like Figure 1 As shown in the figure, the language-guided test-time adaptive zero-shot skeletal action recognition method includes four processes: preprocessing, prompt generation, multi-feature enhancement, and confidence-weighted selection.

[0043] Preprocessing includes the following steps:

[0044] Step 1: A skeleton graph contains M human actions represented by V joints. The input skeleton sequence is randomly sampled with T frames and C channels as the skeleton graph. Its corresponding true label set The datasets used in this example are NTU-RGB+D and PKU-MMD, which record actions using 25 and 22 joints, respectively. To increase recognition speed, this example samples 64 frames from the sequence as input.

[0045] Step 2: Load the target baseline model as a feature extractor / mapper and freeze the parameters. This example uses Shift-GCN as the skeleton feature extractor. CLIP as a text feature extractor VAE as a Stable Feature Mapper This step can obtain the bone feature f s and text features f t :

[0046]

[0047] The generation of prompt words for language guidance includes the following steps:

[0048] Step 3: Use text features as anchors in high-dimensional space for guidance and skeleton features as offsets in high-dimensional space Get the prompt word pt:

[0049]

[0050] The key optimization goals for implementing cue word adaptation during testing to mitigate the negative impact of distribution shift on zero-shot generalization are:

[0051]

[0052] Language-guided multi-feature enhancement consists of the following steps:

[0053] Step 4: Use text features as guidance information to make up for the lack of object information and mix them with skeleton features to obtain a language-guided enhanced view

[0054]

[0055] The parameter α that determines the importance of text guidance c It is randomly generated in the experiment and the interval is (0,1).

[0056] Step 5: Generate a normal distribution ∈ with mean μ and variance σ as Gaussian noise to obtain Aug noise ; Randomly generate a s 0-1 matrices M of the same shape for f s Mask to get Aug mask . and the one obtained in step 4 Composing rich homosemantic skeleton enhanced view sequences Aug:

[0057]

[0058] Aug mask =f s⊙M,

[0059]

[0060] Confidence-weighted selection involves the following steps:

[0061] Step 6: Sample Belong to category The probability calculation formula is:

[0062]

[0063] in, is the frozen stability mapper, sim(·) represents the calculation of cosine similarity, and the hyperparameter temperature coefficient τ is set to 0.9.

[0064] Step 7: Setting up the Indicator With a threshold θ, views below the threshold will not be selected.

[0065] Step 8: Calculate the self-entropy H(p i ) and sent to the indicator to determine whether to filter. The selected view will be weighted according to the confidence level.

[0066]

[0067] Step 9: Update the optimization target of the prompt word in step 3 to:

[0068]

[0069] A zero-shot skeletal action recognition system based on language-guided test-time adaptation includes: a frozen feature extractor / mapper, a language-guided multi-feature generation module, a language-guided prompt word generation module, and a confidence-weighted selection module. During the test phase, the frozen feature extractor / mapper refers to the encoders in the frozen text-image contrastive learning architecture CLIP, CTR-GCN, and variational autoencoder, which are used as text feature extractors, skeletal feature extractors, and noise-removing mappers, respectively. The language-guided multi-feature generation module is used to fuse text features as auxiliary guiding information with skeletal features to obtain prompt words that can be adaptively distributed during testing; text features are used to guide skeletal features to generate more differentiated homosemantic views through feature fusion, which to some extent makes up for the lack of auxiliary information of the skeletal sequence on the object; at the same time, randomly generated Gaussian noise, randomly generated mask matrices, and randomly cropped partial text features of known categories are mixed with skeletal features to obtain several homosemantic skeletal enhanced view sequences; Gaussian noise and mask strategies are used to generate richer enhanced views Augmentednoise and Aug mask The language-guided prompt generation module regards text features as the starting point for generating prompt words in a high-dimensional feature space and skeleton features as the required displacement. The two work together to obtain fine-tunable adaptive prompt words, bridging the semantic space gap between text and skeleton modalities. During the test, the confidence-weighted selection module judges the confidence of the prediction results of the view based on the self-entropy of the prediction results under different views during the process of prompt word fine-tuning. By giving higher weights to give priority to predictions with higher confidence and excluding predictions with low confidence, it can effectively alleviate the problem of severe distribution shift.

[0070] Tables 1 and 2 detail the zero-shot partitioning strategy for the NTU-RGB+D dataset.

[0071] Table 1 General zero-shot partitioning of the NTU-RGB+D 60 dataset

[0072]

[0073] Table 2 General zero-shot partitioning of the NTU-RGB+D120 dataset

[0074]

[0075] Table 3 compares the results of our method with other methods under four common visible / invisible split settings for the NTU-RGB+D dataset. On the NTU-RGB+D 60 dataset, the proposed method uses a 55 visible class / 5 invisible class (55 / 5) and a 48 visible class / 12 invisible class (48 / 12) split scheme. On the NTU-RGB+D 120 dataset, the proposed method uses a 110 visible class / 10 invisible class (110 / 10) and a 96 visible class / 24 invisible class (96 / 24) split scheme. Top-1 accuracy is used for quantitative evaluation, with the best results in bold. The proposed method demonstrates significant advantages across all experiments: Under both split schemes on the NTU-60 dataset, the proposed method achieves 2.4% and 1.08% improvements over the second-best performing SA-DVAE, respectively. On the more challenging NTU-120 dataset, this advantage further increases to 3.6% and 1.24% performance gains, demonstrating the effectiveness of our method.

[0076] Table 3 Comparative experimental results of the present invention and current advanced methods under universal zero-shot partitioning

[0077]

[0078]

[0079] Table 4 shows the average results of the present invention under three random visible / invisible split settings (55 / 5, 110 / 10 and 46 / 5) on NTU-RGB+D 60, NTU-RGB+D 120 and PKU-MMD, respectively. The Top-1 accuracy is used for quantitative evaluation, and the influence of category division bias on the experimental results is eliminated by calculating the average performance index. The bold results are the best results. Even in the experiments on the most challenging NTU-120 dataset (with the largest sample size and the largest number of invisible categories), the present invention has achieved significant average performance improvements. For example, compared with the suboptimal method, the present invention achieved an average accuracy improvement of 2.77% in the experiments on the NTU-120 dataset. In the experiments on the NTU-60 and PKU-MMD datasets, the performance gains also reached 1.85% and 1.43%, respectively. This once again verifies the effectiveness of the proposed method in the zero-sample skeletal action recognition task.

[0080] Table 4 Comparative experimental results of the present invention and the current advanced method under random zero-sample partitioning

[0081]

[0082] Table 5 is an ablation experiment on the effectiveness of the three modules of the present invention, and the results in bold are the optimal results. Among them, LGPG, LA-MFA and CWS represent the language-guided prompt word generation module, the language-guided multi-feature enhancement module and the confidence-weighted selection module, respectively. The method in which the three modules work together shows the best performance. Regardless of which module is disabled, the final zero-sample recognition accuracy will be reduced. The experiment with the largest change in experimental results is the experiment in which the LGPG module is disabled on the NTU-120 (110 / 10) dataset. Its Top-1 accuracy is reduced by 60%, and the average performance of the four groups of experiments has decreased by 43%. Not using language information to assist in the generation of enhanced views will also weaken the zero-sample generalization ability of the model, and the average performance of the four groups of experiments has decreased by 6%.

[0083] Table 5 Ablation experiments on the effectiveness of the three modules proposed by the present invention

[0084]

[0085] Figure 3This is the confusion matrix of the prediction results of the present invention and the current optimal method under the setting of 10 invisible classes in the NTU-RGB+D 120 dataset. It can be found that the 106th category "swinging a knife" is very easy to be misclassified as the 49th category "swinging a fist". This is because in the absence of object information, the skeletal performance of "swinging a knife" and "swinging a fist" are very similar, and the semantics can be expressed as "swinging both hands in front". Similarly, the 37th category "salute" is very easy to be misclassified as the 43rd category "headache", but the present invention (left side) has improved the recognition accuracy of this type of action to a certain extent. In addition, the lack of object information leads to a more serious distribution shift problem. The previous method (right side) cannot generalize the two actions of "swinging a fist" and "playing table tennis", and the present invention has made up for this defect to a large extent.

[0086] Obviously, those skilled in the art should understand that, each step of the zero-sample skeleton action recognition method of the test time self-adaptation based on language instruction of the above-mentioned embodiment of the invention or each module of the zero-sample skeleton action recognition system of the test time self-adaptation based on language instruction can be realized with general computing device, they can concentrate on single computing device, or be distributed on the network that a plurality of computing devices are formed, alternatively, they can be realized with the executable program code of computing device, thereby, they can be stored in the storage device and be executed by computing device, and in some cases, can be different from the step shown or described in the order performed herein, or they are made into each integrated circuit module respectively, or a plurality of modules or step therein are made into single integrated circuit module and realize.Like this, the embodiment of the invention is not restricted to any specific hardware and software combination.

Claims

1. A language-guided test-time adaptive zero-shot skeletal action recognition method, characterized in that: The steps include: Step 1: Randomly sample several frames from the skeleton sequence of the unseen category in the test phase as test samples; Step 2: Load the target baseline model as a feature extractor / mapper and freeze the parameters; Step 3: Extract the skeleton features and text features of the current test sample; Step 4: The text features are used as auxiliary guiding information and fused with the skeleton features to obtain prompt words that can be adaptively distributed during testing; Step 5: Mix the randomly generated Gaussian noise, the randomly generated mask matrix, and the randomly cropped text features of known categories with the skeleton features to obtain several skeleton-enhanced view sequences with the same semantics. Step 6: Stability mapping between the cue words and the skeleton augmented view sequence; Calculate the similarity between the text features of the prompt word and the skeletal features of the action sequence as the classification basis, and obtain the probability distribution of each unseen category prediction under all enhanced views; Step 7: Assign weights to the probability distribution of each bone-enhanced view output on the unseen category according to the size of the self-entropy value, from low to high, to obtain the weighted entropy; The predictions whose entropy values exceed the set threshold are ignored, and the weights of the retained views sum to 1; Step 8: Optimize the adaptive distribution of prompt words based on minimization of weighted entropy; Step 9: Output the current classification result.

2. The language-guided test-time adaptive zero-shot skeletal action recognition method according to claim 1, characterized in that: Given a set of human motion skeleton sequences This includes the visible subset and the invisible subset and They are the corresponding visible subsets and the invisible subset The tag set of The goal of zero-shot skeletal action recognition is to With the corresponding visible label Train and predict the input sequence Category The zero-shot skeletal action recognition method directly realizes the recognition of unseen categories in the testing phase based on fine-tunable cue words.

3. The method for zero-shot skeletal action recognition based on language guidance and test time adaptation according to claim 1, wherein: The process of extracting skeleton features and text features in step 3 is: in, and They represent the frozen skeleton feature extractor and text feature extractor respectively, x is the current input sample, superscripts s and t represent the skeleton mode and text mode respectively, f s is the bone feature, f t is a text feature.

4. The language-guided test-time adaptive zero-shot skeletal action recognition method according to claim 1, characterized in that: The prompt word pt in step 4 is anchored by text features as guidance information, and the skeleton features of the sample are used as displacement in the feature space. get:

5. The language-guided test-time adaptive zero-shot skeletal action recognition method according to claim 3, characterized in that: In step 4, the prompt words are optimized. The optimization goals are: in, represent and pt is a hint word that is expected to be optimized and can adapt to the distribution during testing.

6. The language-guided test-time adaptive zero-shot skeletal action recognition method according to claim 1, characterized in that: The generation method of the skeleton enhanced view sequence Aug in step 5 is as follows: Aug mask =f s ⊙M, Among them, Aug noise is the enhanced view obtained by mixing random Gaussian noise, Aug mask is the enhanced view obtained by applying the random mask strategy, is an enhanced view of language guidance obtained by randomly cropping the text features of the seen category c and mixing them with the current test skeleton sequence; ⊙ represents the operation of the Hadamard product, M is a randomly generated 0 / 1 matrix; ∈ is a normal distribution with mean μ and variance σ; α c It is a random number (0,1) that determines the text feature corresponding to category c For the current sample bone feature f s degree of guidance.

7. The language-guided test-time adaptive zero-shot skeletal action recognition method according to claim 6, characterized in that: After obtaining the skeleton enhanced view sequence Aug, the optimization objective of step 4 is updated to: Among them, Aug j is the jth of all K skeleton-enhanced view sequences, the base of log is 2, is the probability of the jth bone-enhanced view and the prompt word being the i-th category, sample Belong to category The probability calculation formula for i=1,2,…,N is: in, is the frozen stability mapper, sim(·) represents the computation of cosine similarity, and τ is the hyperparameter temperature coefficient.

8. The language-guided test-time adaptive zero-shot skeletal action recognition method according to claim 1, characterized in that: Based on the similarity, the probability distribution p of each bone-enhanced view predicted on the unseen category is obtained i ; The way to calculate the weighted entropy in step 7 is: Among them, H represents the probability distribution p i Calculate the self-entropy. Represents an indicator. Prediction results with confidence lower than θ will be ignored, and the prediction results corresponding to the retained i-th view will also be weighted according to the confidence 9. A language-guided test-time adaptive zero-shot skeletal action recognition system, characterized in that: include: Frozen feature extractor / mapper, language-guided multi-feature generation module, language-guided prompt word generation module, and confidence-weighted selection module; The frozen feature extractor / mapper is used to extract skeleton features and text features from skeleton sequence samples; The language-guided multi-feature generation module is used to fuse text features as auxiliary guiding information with skeleton features to obtain prompt words that can be adaptively distributed during testing; randomly generated Gaussian noise, randomly generated mask matrices, and randomly cropped partial text features of known categories are mixed with skeleton features to obtain several semantically homogeneous skeleton-enhanced view sequences; the language-guided prompt generation module regards text features as the starting point for generating prompt words in a high-dimensional feature space and skeleton features as the required displacement, and the two work together to obtain fine-tunable adaptive prompt words; the confidence weighted selection module judges the confidence of the prediction results under different views based on their self-entropy during the test time adaptation process, giving priority to predictions with higher confidence and excluding predictions with low confidence by assigning higher weights.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the language-guided test-time adaptive zero-sample skeletal action recognition method according to any one of claims 1 to 8.

Images