A long tail skeleton action recognition method based on shapley value guided data mixing

Abstract

The application discloses a long-tail skeleton action recognition method based on Shapley value guided data mixing, and belongs to the fields of action recognition and long-tail learning. The application calculates the saliency of different joint parts in human skeleton data through Shapley value, trims the saliency motion region of the tail category, and performs data mixing enhancement. The application solves the problem of low skeleton action recognition accuracy under long-tail distribution, expands the data samples of the tail category, and promotes the decision boundary learning of the tail category. The application updates the saliency estimation result through online momentum, generates more representative mixed data, and optimizes and trains the model by using the reweighted loss of long-tail learning, thereby significantly improving the recognition effect of the tail category and the generalization ability of the model.

Description

Technical Field

[0001] This invention belongs to the fields of action recognition and long-tail learning, and specifically designs a long-tail skeleton action recognition method based on Shapley Value-guided data mixing. Background Technology

[0002] Human skeleton-based action recognition refers to the technique of inferring and classifying different actions using the three-dimensional coordinate data of various joints in the human body. In the past, research mainly focused on modeling skeleton data using recurrent neural networks (RNNs) and convolutional neural networks (CNNs), treating it as a time series or pseudo-two-dimensional image. However, with a deeper understanding of the topological relationships of human structure, methods based on graph convolutional neural networks (GCNs) have gained attention and achieved significant performance improvements. GCNs can effectively capture the relationships between nodes in skeleton data, thus better understanding the structure and contextual information of actions. In recent years, Transformer-based models have also shown potential in the field of action recognition. By introducing a self-attention mechanism, Transformer models can capture long-term dependencies in skeleton sequences, thereby improving the accuracy and performance of action recognition.

[0003] Long-tail learning refers to the process of a model learning a good representation space, especially for the few tail categories, when there is a severe imbalance in the dataset. The challenge in this scenario lies in the fact that with a long-tail distribution, the model tends to learn more frequently occurring categories while underlearning less frequent categories, leading to a skewed representation space. Common methods to address long-tail learning include resampling and reweighting. Resampling methods handle data imbalance by sampling tail categories or undersampling head categories, while reweighting methods balance gradient flow by assigning higher weights to less frequent categories. Recently, extensive research has been conducted on solutions to the image long-tail learning problem, including data augmentation, ensemble learning, decoupled learning, and contrastive learning. These methods employ more advanced techniques to provide new perspectives and approaches for solving the image long-tail learning problem.

[0004] However, considering the naturally long-tailed distribution of human movements, current skeleton motion recognition methods are all designed based on balanced datasets, leading to severe performance degradation under real-world long-tailed data. Furthermore, existing long-tail learning methods primarily target image data, while the long-tail problem of skeleton data has not been fully explored. In general, the challenges of long-tail learning in skeleton motion recognition tasks mainly include: 1) Spatial sparsity and compactness of data: Skeleton data is typically sparse and compact, lacking background information, while its key spatial motion patterns are more prominent; 2) Additional temporal information: Skeleton data possesses an additional temporal dimension compared to image data, and its temporal dynamic information is crucial for motion recognition. These characteristics mean that directly applying long-tail learning methods from image data to skeleton data leads to suboptimal results, and the overall performance cannot meet the needs of practical applications. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a long-tail skeleton action recognition method based on Shapley value-guided data mixing. By cropping out the salient motion region of the tail category, the method obtains a mixed enhancement result with different data, thereby expanding the data sample of the tail category and promoting the learning of the decision boundary of the tail category.

[0006] The technical solution adopted in this invention is as follows:

[0007] A long-tail skeleton action recognition method based on Shapley value-guided data mixing includes the following steps:

[0008] Human skeleton data with action category labels is used as the training dataset;

[0009] Human skeleton data is divided into five body parts: left arm, right arm, left leg, right leg, and torso. The model predicts the action category, and the significance score of different body part combinations under the action category is estimated based on the Shapley value, resulting in a list of significance estimates for different body part combinations.

[0010] Two human skeleton data points, s1 and s2, are sampled. The number of samples of each action category in the training dataset is compared. For the human skeleton data point s1 with fewer samples, its significance score is obtained from the significance estimation list. Sampling is performed based on the significance score, and the sampled body parts are combined as the body parts to be mixed.

[0011] Two human skeleton data sets, s1 and s2, are mixed. The body parts to be mixed in human skeleton data s1 are combined with the remaining body parts of the corresponding body parts in human skeleton data s2 to obtain enhanced mixed data.

[0012] The unmixed data without enhancement and the enhanced mixed data are input into the model separately, and the model is trained iteratively. The cross-entropy loss and the reweighted loss of any long-tail learning method are calculated separately. The sum of the two losses is the total loss.

[0013] The trained model is used to predict the action category of the human skeleton data to be predicted, and the action category prediction results are output.

[0014] Furthermore, the model employs a channel-wise topology refinement graph convolution network model, CTR-GCN (Channel-wise Topology Refinement Graph Convolution Network).

[0015] Furthermore, the significance score is calculated as follows:

[0016]

[0017] Where U is the complete set of body parts, b is the combination of body parts whose significance scores are to be estimated, and r is a subset of (Ub); f c (·) is the score of action category c output by the model, comb(·) is the function to calculate the number of body part combinations, and |·| is the operation to calculate the cardinality of the set.

[0018] Furthermore, in each iteration, the model randomly selects a combination b of body parts and estimates the significance score. The significance scores in the significance estimate list are then updated using a moving exponential average.

[0019] Furthermore, the sampling steps based on this significance score include:

[0020] significance score The L1 normalization and softmax operations are performed as follows:

[0021]

[0022] The formulas for functions norm and softmax are: τ is the temperature coefficient, and |b| is the number of skeletal nodes contained in the body part combination b.

[0023] The result calculated by the above formula is used as a probability distribution for sampling.

[0024] Furthermore, the formula for calculating cross-entropy loss is:

[0025]

[0026] Among them, L ceThe cross-entropy loss is denoted by s, where s is the score of the different classes output by the model. i Let N be the score for the i-th category. c Let y be the total number of action categories, y be the initial action label for each category, and n be the total number of action categories. i Let be the number of examples of the i-th action category in the training dataset.

[0027] Furthermore, the formula for calculating the reweighted loss is:

[0028]

[0029] Among them, L balance The loss is a reweighted loss, where s is the score of the different classes output by the model. i Let N be the score for the i-th category. c Let y be the total number of action categories, y be the label of the mixed data, and n be the total number of action categories. j denoted as the number of examples of the j-th action category in the training dataset.

[0030] Furthermore, the labels for the mixed data are obtained by linear interpolation of the action categories, with the mixing ratio λ used as the weight for different action categories.

[0031] Compared with the prior art, the positive effects of the present invention are as follows:

[0032] 1. Precise Mixing and Efficient Estimation Based on Saliency Regions: The saliency of different joint sites is calculated using Shapley values. Saliency regions of tail category samples are retained for mixing, resulting in more representative mixed data and facilitating the learning of decision boundaries for tail categories. Momentum updates are used to continuously update the saliency estimation results online, maintaining a saliency list for different body parts for each action category, ensuring the real-time nature and accuracy of the saliency estimation.

[0033] 2. Enhanced Data Diversity and Model Optimization: A data fusion enhancement strategy generates more diverse data samples, enriching the training dataset and enhancing the model's generalization ability. Normalization and softmax operations are performed based on the saliency estimation results, and sampling is conducted using saliency as the probability distribution to ensure the quality and diversity of the fused data. Balanced Softmax loss is used for optimized training on the fused data, combined with cross-entropy loss from the unmixed data, enabling the model to better adapt to long-tailed data distributions and improving overall recognition performance.

[0034] 3. Improve the recognition accuracy of tail categories: By cropping the salient motion regions of tail categories and performing data mixing enhancement, the data samples of tail categories are expanded, which significantly improves the recognition effect of tail categories. Attached Figure Description

[0035] Figure 1This is the overall training flowchart used in the embodiments of the present invention.

[0036] Figure 2 This is a flowchart illustrating the specific reasoning and usage stages of this invention. Detailed Implementation

[0037] To make the various technical features, advantages, and effects of the present invention more apparent and understandable, a detailed description is provided below in conjunction with the accompanying drawings. It should be noted that the hyperparameter settings given in the following embodiments are merely preferred implementations and are not intended to limit the scope. Those skilled in the art can select the appropriate number and set certain layers according to actual needs, which should be understood.

[0038] This embodiment discloses a long-tailed skeleton action recognition method based on Shapley value-guided data mixing, such as... Figure 1 As shown, given a human skeleton data sample, the saliency of different joint locations is calculated using the Shapley value formula. During data fusion enhancement, the saliency regions of the tail category samples are preserved, resulting in more representative fused data for the tail category, thus facilitating the CTR-GCN model's learning of the decision boundary for the tail category. This model is trained for action recognition tasks using the proposed Shapley value-guided data fusion strategy. The specific processing steps of this method are explained below:

[0039] Step 1: Collect and build a large amount of human skeleton data, and label the action categories to form a training dataset.

[0040] Step 2: Based on the Shapley value, perform online saliency estimation for any human skeleton data in the training dataset. Specifically, the human skeleton data is divided into five body parts: left arm, right arm, left leg, right leg, and torso. The model predicts the action category, and the saliency of different body part combinations under that action category is estimated based on the Shapley value. In each training round, the saliency estimation results are continuously updated online using a momentum-based update method, maintaining a saliency list for different body parts for each action category, as shown in Table 1. The values ​​in the table are the saliency scores of body part combinations under the corresponding action category.

[0041] For example, under action category c, the salience score of body part combination b, such as waving hands, is defined as...

[0042]

[0043] Where U is the complete set of body parts, i.e., {left arm, right arm, left leg, right leg, torso}, b is the combination of body parts whose saliency is to be estimated, such as {right arm, torso}, and r is a subset of (Ub); f c(·) represents the score of action category c output by the model, comb(·) is the function to calculate the number of body part combinations, and |·| is the operation to calculate the cardinality of the set. In each iteration, a body part combination b is randomly selected and estimated, and then updated using a moving exponential average.

[0044] Table 1. List of significance estimates

[0045]

[0046] Step 3: Sample two human skeleton data sets and obtain the body parts to be mixed. Specifically, for the two sampled human skeleton data sets s1 and s2, assuming that the action category to which s1 belongs has fewer samples in the training dataset, first obtain the saliency estimate list of the action category to which s1 belongs obtained in step 2), normalize the result, and use it as the probability distribution to sample the mixed parts. The more significant a part is in the action category, the higher its probability of being sampled.

[0047] In this embodiment, assuming the sampled human skeleton data belongs to two action categories—waving and standing—the number of samples for each action category in the training dataset is first compared. Assuming the waving category is rarer, the estimated value from step 2) is obtained. Then perform L1 normalization and softmax operation on it:

[0048]

[0049] The formulas for functions norm and softmax are: τ is the temperature coefficient, and |b| is the number of skeleton nodes contained in the combination b. The result obtained from the above formula is used as a probability distribution for sampling. Assume the sampled b... sample The right arm and torso are the body parts that will be mixed.

[0050] Step 4: Perform data blending to obtain enhanced mixed data. Specifically, for the human skeleton data s1 and s2 sampled in step 3), blend the body parts sampled in step 3), retain the body parts corresponding to s1 data, and replace the body parts corresponding to s2 data to obtain the blended skeleton data s. m .

[0051] In this embodiment, the waving and standing skeleton data sampled in step 3) are mixed according to the parts sampled in step 3), that is, the {right arm, torso} parts of the waving category data are retained, while the remaining body parts of the standing category data are retained, and the enhanced mixed data s is obtained by combining them. m .

[0052] Step 5: Perform one round of model training. Specifically, input the unmixed data (without enhancement) and the mixed data (with the enhancements described above) into the model for separate training. The model uses cross-entropy loss L... ce And the reweighted loss L of any long-tail learning balance Training is performed, for example, using the existing Balanced Softmax loss. For unmixed input data, the initial action class is used as the label, and cross-entropy loss is used for optimization. For mixed input data, the linear interpolation result of the action class is used as the label. Assuming the mixing ratio is λ, the label of the mixed data is: waving * λ + standing * (1 - λ), and Balanced Softmax loss is used for optimization.

[0053] The model's total loss function is:

[0054] L = L ce +L balance

[0055]

[0056] Where s represents the scores of different categories output by the model, s i That is, the score of the i-th category, N c This represents the total number of action categories, y represents the corresponding label, and n represents the total number of action categories. j This represents the number of examples of the j-th action category in the training dataset. Training ends when the number of model iterations reaches a preset value.

[0057] Step 7: In the inference phase, the human skeleton data to be predicted is input into the trained model for action category prediction. The model outputs the action category prediction result, such as... Figure 2 As shown.

[0058] Experimental test:

[0059] The baseline model, using the method of this invention, was experimentally tested on the long-tailed NTU-60 skeleton action recognition benchmark set. The experimental results show that the recognition accuracy can be improved from 70.4% to 80.8%; on the long-tailed NTU-120 skeleton action recognition benchmark set, the recognition accuracy can be improved from 64.2% to 73.0%. It can be seen that this invention significantly improves the skeleton action recognition results under long-tailed distribution.

[0060] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A long-tail skeleton action recognition method based on Shapley value-guided data mixing, characterized in that, Includes the following steps: Human skeleton data with action category labels is used as the training dataset; Human skeleton data is divided into five body parts: left arm, right arm, left leg, right leg, and torso. The model predicts the action category, and the significance score of different body part combinations under the action category is estimated based on the Shapley value, resulting in a list of significance estimates for different body part combinations. Sample two human skeletons and Compare the number of samples in the training dataset for each action category of these two human skeleton datasets. For the human skeleton dataset with fewer samples... Obtain the significance score from the significance estimation list, sample based on the significance score, and combine the sampled body parts as the body parts to be mixed; Two human skeleton data and Data mixing is performed, including human skeleton data. Body parts to be mixed with human skeleton data The remaining body parts corresponding to the body parts are combined to obtain the enhanced mixed data; The unmixed data without enhancement and the enhanced mixed data are input into the model separately, and the model is trained iteratively. The cross-entropy loss and the reweighted loss of any long-tail learning method are calculated separately. The sum of the two losses is the total loss. The trained model is used to predict the action category of the human skeleton data to be predicted, and the action category prediction results are output.

2. The method as described in claim 1, characterized in that, The model uses the channel-aware convolutional neural network model CTR-GCN.

3. The method as described in claim 1, characterized in that, The significance score is calculated as follows: in, It is a complete collection of body parts. b The combination of body parts whose significance scores are to be estimated. r yes A subset of; It is the action category output by the model. c The score, It is a function that calculates the number of combinations of body parts. It is an operation for calculating the cardinality of a set.

4. The method as described in claim 1 or 3, characterized in that, In each iteration, the model randomly selects a body part for combination. b And estimate the significance score. Then, the significance scores in the significance estimate list are updated using a moving exponential average.

5. The method as described in claim 1, characterized in that, The formula for calculating cross-entropy loss is: in, For cross-entropy loss, The scores for different categories output by the model. Let be the score for the i-th category. The total number of action categories. For the initial category action label, Let be the number of examples of the i-th action category in the training dataset.

6. The method as described in claim 1, characterized in that, The formula for calculating reweighted loss is: in, For weighted loss, The scores for different categories output by the model. Let be the score for the i-th category. The total number of action categories. Labels for mixed data, denoted as the number of examples of the j-th action category in the training dataset.

7. The method as described in claim 6, characterized in that, The labels for the mixed data are obtained by linear interpolation of action categories, using a mixing ratio. This serves as the weight for different action categories.

Images