Course learning based multi-quality skeletal sequence action recognition method

Abstract

The application discloses a multi-quality skeleton sequence action recognition method based on course learning, which comprises the following steps: collecting a large number of videos containing human body actions and having action category labels, and dividing the videos into a training set and a test set; extracting the coordinate information of the human body joint points of each frame of image in the video to generate a continuous skeleton sequence; performing data enhancement on the skeleton sequence; constructing a neural network model for action recognition, then inputting the data of the training set into the neural network model in batches, training the neural network model until a preset number of rounds is reached, wherein during the process of training the model, the prior knowledge and the posterior knowledge are dynamically adjusted to evaluate the sample difficulty, in addition, the concept of learning gravity is introduced, and the learning gravity gradually transitions from simple samples in the early stage to difficult samples at different stages. The method provided by the application can improve the recognition efficiency and the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition using computer vision, and in particular to a multi-quality skeleton sequence action recognition method based on curriculum learning. Background Art

[0002] The primary goal of action recognition is to classify data containing one or more people performing specific actions. Currently, a wide variety of action recognition modalities are commonly used, including visual modalities such as RGB video, skeleton sequences, depth data, infrared sequences, point clouds, and event streams, as well as invisible modalities such as audio, velocity, radar signals, and Wi-Fi signals. Each modality has its own unique advantages and disadvantages. Skeletal sequences, in particular, have a significant advantage in action recognition. Their simplicity and effectiveness, robustness to viewpoint shifts, and independence from background have made them a focal point of action recognition research.

[0003] Currently, research on high-quality skeleton sequences has achieved excellent recognition performance. However, in real-world applications, the cost of obtaining high-quality skeleton sequences is often quite high. Due to technical and cost limitations, many devices can only use lightweight pose estimation networks to extract low-quality skeleton sequences from videos. When these low-quality skeleton sequences are input into models trained on high-quality skeleton sequences, the accuracy is significantly reduced. Moreover, current research on low-quality skeletons generally does not scale well to high-quality skeleton sequences, which poses a significant challenge to practical applications.

[0004] Previous methods were mostly trained and evaluated based on relatively clean skeleton data. In recent years, a growing number of studies have focused on how to handle action recognition from noisy skeletons. These studies consider this problem as a challenge for skeleton denoising, first using linear or nonlinear transformations to filter out skeleton noise before proceeding to action recognition. Some methods, based on biologically constrained skeletal structure priors, use linear transformations for skeleton denoising. However, relying solely on linear transformations based on human body priors may not be optimal and may not be sufficiently effective in removing noise. To overcome the shortcomings of linear transformations, PE-GCN proposed a nonlinear denoising transformation method based on autoencoders. However, this method is problematic for training by simply enforcing a mean squared loss between the noisy input and the reconstructed output, as it can easily lead the model to fall into a trivial solution (i.e., the same mapping). Furthermore, the unknown nature of the skeleton degradation process makes it challenging to find a universal denoising model and further promote action recognition.

[0005] The backbone GCN of the RA-GCN network consists of a series of ordered streams that are responsible for learning discriminative features from previously inactivated joints, and only the information of inactivated joints will be passed to the next stream. By activating some unimportant joints multiple times, the network is not only dependent on certain joints, thereby enhancing the robustness of the network. Other methods use predictive coding to train the model by maximizing the mutual information between normal skeletons and noisy skeletons. The knowledge distillation method uses high-quality human skeleton sequence data to pre-train the teacher model, and then uses the pre-trained teacher model to guide the training of the student model with low-quality human skeleton sequence data. MaskCLR improves robustness by masking the high-attention joints and forcing the network to learn the motion characteristics of the whole body.

[0006] Action recognition based on skeletal data has recently performed well on various datasets. However, due to the varying quality of skeletal data extracted in real-world applications, this can lead to a sharp decline in performance. A simple solution is to train the network on as many skeletal data of varying quality as possible, thereby enabling it to learn discriminative features of varying skeletal quality. Therefore, it is crucial to find ways to better adapt existing methods to learning from skeletal data of varying quality.

[0007] Therefore, proposing a new action recognition method for multi-mass human skeleton sequences has become an urgent problem to be solved. Summary of the Invention

[0008] In view of this, the present invention provides a multi-mass skeleton sequence action recognition method based on curriculum learning to achieve accurate action recognition.

[0009] The technical solution provided by the present invention is specifically a multi-mass skeleton sequence action recognition method based on curriculum learning, comprising the following steps:

[0010] S1: Collect a large number of RGB videos containing human actions and with action category labels, and divide the collected videos into training and test sets;

[0011] S2: Extract the coordinate information of the human joint points in each frame of the video and generate a continuous bone sequence according to the bone structure connection method;

[0012] S3: performing data enhancement on the skeleton sequence to obtain a multi-quality skeleton sequence;

[0013] S4: constructing a neural network model for action recognition, then inputting the training set data into the neural network model in batches, and training the neural network model until a preset number of rounds is reached;

[0014] The process of training the model using any batch B of data is as follows:

[0015] S41: Calculate the comprehensive difficulty of the sample using prior knowledge and a posteriori knowledge, where the a posteriori knowledge is the model prediction accuracy and the prior knowledge is the average confidence of all relevant nodes in the skeleton sample estimated using the posture estimation algorithm. The calculation formula for the comprehensive difficulty of the sample is as follows:

[0016] s=α×P t +β×Q conf ;

[0017] In the formula, s represents the comprehensive difficulty of the sample. The larger the value, the simpler the sample. α is the weight of the posterior knowledge. P t is the probability that the model predicts correctly, β is the weight of prior knowledge, Q conf It is the average confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm. As the number of training rounds increases, the weight of prior knowledge decreases, and the weight of posterior knowledge increases.

[0018] S42: Calculate the learning center of gravity at the current stage, and use the learning center of gravity and the comprehensive difficulty of the sample to calculate the sample loss weight. Then, use the sample loss weight to weight the cross entropy loss function to obtain the final loss;

[0019] Among them, as the number of training rounds increases, the learning focus gradually shifts from simple samples to difficult samples;

[0020] The loss weights of different samples are calculated using the analytical expression of the chi-square distribution, where the learning center of gravity is used as the degree of freedom of the chi-square distribution. The loss weight of the sample is calculated as follows:

[0021]

[0022] Among them, w represents the loss weight of the sample, k represents the learning center of gravity, s represents the comprehensive difficulty of the sample, the larger s is, the simpler the sample is, and Γ represents the Gamma function;

[0023] S43: Update the neural network model using the final loss.

[0024] Preferably, in S3, the specific steps of performing data enhancement on the skeleton sequence are as follows: performing a random occlusion operation on the skeleton sequence and adding random noise to the coordinates of the joint points in the skeleton sequence, wherein each occlusion randomly selects some joint points in the skeleton sequence, and the occlusion ratio is uniformly randomly taken between 10% and 30%. For the occluded joint points, their coordinate values ​​are set to 0, and the selection of the occlusion area follows the principle of uniform distribution. The added noise is Gaussian noise with a mean of 0 and a standard deviation uniformly randomly taken between 0.1 and 0.3.

[0025] Further preferably, in S41, the calculation formula of the weight of the posterior knowledge is as follows:

[0026]

[0027] Where α is the weight of the posterior knowledge, epoch cur Indicates the number of rounds of the current epoch, epoch total Indicates the total number of epochs required for training;

[0028] The calculation formula of the weight of the prior knowledge is as follows:

[0029] β=1-α;

[0030] Among them, β is the weight of prior knowledge.

[0031] Further preferably, in S41, the average confidence value Q of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm is conf The calculation formula is as follows:

[0032]

[0033] Where q tvm Represents the confidence of the t-th frame, the v-th joint point, and the m-th person. T represents the total number of frames of the sample, V represents the total number of joints in the sample, and M represents the total number of people in the sample.

[0034] Further preferably, in S42, the calculation formula of the learning center of gravity is as follows:

[0035]

[0036] Among them, k represents the learning center, A and B are hyperparameters used to set the size of the initial and final values, and epoch cur Represents the current epoch round number, epoch total Represents the total number of epochs required for training.

[0037] More preferably, S42 further includes the following steps:

[0038] The loss weights of the samples in batch B are normalized using a normalization method. The formula for normalizing the loss weights of the samples in batch B to a certain interval is as follows:

[0039]

[0040] Among them, w norm is the normalized sample loss weight of the batch B, w min and w max It is a manually set hyperparameter, representing the maximum and minimum values ​​of the sample loss weight, batch min and batch max They represent the maximum and minimum values ​​of the comprehensive difficulty of the samples in batch B, respectively, and w is the loss weight of the sample.

[0041] Further preferably, in S42, the cross entropy loss function is weighted using the normalized loss weight of the batch B samples to obtain the final loss, wherein the formula for weighting the cross entropy loss function is as follows:

[0042] Loss=w norm ×Loss CE ;

[0043] Among them, Loss is the final loss of the batch B, w norm is the normalized sample loss weight of batch B, Loss CE is the cross entropy loss function used to train classification neural networks.

[0044] The multi-mass skeleton sequence action recognition method based on curriculum learning provided by the present invention has the following advantages:

[0045] 1, the present invention innovatively utilizes priori and a posteriori knowledge to evaluate bone sample simultaneously, and the weight of priori and a posteriori knowledge can change dynamically according to training process. In the initial stage of training, model is limited to the understanding of data, and now a posteriori knowledge (such as model prediction accuracy) reliability is lower, so a priori knowledge (bone quality) weight is relatively high, mainly relies on bone quality to guide model learning to be relatively simple and the sample with guaranteed quality, avoids model from producing wrong learning direction because of over-reliance on unreliable a posteriori knowledge. Along with the advancement of training, model is gradually deepened to the learning of data, and the reliability of a posteriori knowledge strengthens, and its weight increases accordingly, and a priori knowledge weight then reduces gradually. This dynamic adjustment mechanism ensures that in different training stages, sample assessment can make full use of the advantage of two kinds of knowledge, avoids simply relying on a posteriori knowledge in the early stage because model is not fully learned and causes unreliable, and a priori knowledge in the later stage because of being unable to reflect the limitation of model actual learning effect.

[0046] 2. The present invention proposes a novel and unique learning center of gravity theory. During the model training process, the learning center of gravity will gradually transition from the early simple samples to the difficult samples as the training stage changes, and its starting and ending positions are accurately determined by reasonably setting the hyperparameters. For samples that are simpler than the learning center of gravity, the loss weight is moderately reduced so that the model does not pay too much attention to these relatively easy-to-master samples during the learning process, thereby allocating more learning resources to more challenging samples. For samples that are more difficult than the learning center of gravity, the loss weight is greatly reduced to avoid the model falling into a local optimal solution due to excessive pursuit of learning these extremely difficult samples, affecting the overall learning efficiency.

[0047] 3, the method proposed by the present invention is particularly suitable for the situation that skeleton quality is diverse and uniform, and this is a major advantage in actual applications. When training model, high-quality data can be utilized to make various degrees of low-quality data by specific data enhancement technology (such as random blocking, adding random noise and other modes, the principle and effect of data enhancement can be briefly described here), and these data are added to training together. This mode can allow model to fully come into contact with the skeleton sequence of various qualities in training process, learn the discriminant characteristics of different quality skeleton data, thereby effectively promote the robustness of model to low-quality skeleton sequence.

[0048] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 A flowchart of a multi-mass skeleton sequence action recognition method based on curriculum learning is provided in an embodiment disclosed by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] In order to solve the problem that current technology cannot efficiently utilize sample information in multi-quality data processing, which leads to limited model performance, this implementation provides a multi-quality skeleton sequence action recognition method based on curriculum learning, see Figure 1 , the method comprises the following steps:

[0054] S1: Collect a large number of RGB videos containing human actions and with action category labels, and divide the collected videos into training and test sets;

[0055] Preferably, the training set and test set are divided in a ratio of 7:3;

[0056] The division process uses a stratified random sampling method to ensure that the distribution ratio of each action category in the training set and the test set is similar, avoiding the impact of data bias on model training.

[0057] S2: Extract the coordinate information of the human joint points in each frame of the video and generate a continuous bone sequence according to a specific bone structure connection method (such as the commonly used 2D human bone structure model);

[0058] The coordinate information of the human body joints can be obtained by processing the RGB video obtained in S1 using a selected posture estimation method. The posture estimation method can be selected as a default posture estimation method from OpenPose, Pifpaf, HRNet, VITPose and other methods that can be used for 2D posture estimation according to the actual application scenario and the requirements for accuracy and speed.

[0059] In the process of generating a continuous skeleton sequence, the joint point coordinates are normalized so that their value range is between [0, 1] to facilitate subsequent calculation and processing.

[0060] S3: performing data enhancement on the skeleton sequence to obtain a multi-quality skeleton sequence;

[0061] The specific steps are as follows: perform a random occlusion operation on the skeleton sequence and add random noise to the coordinates of the joint points in the skeleton sequence to simulate the measurement errors and noise interference that may occur in actual applications. Through the above-mentioned data enhancement method, a multi-quality skeleton sequence is obtained to increase the diversity and complexity of the data and improve the model's generalization ability for skeleton data of different qualities.

[0062] Each occlusion randomly selects a subset of joints in the skeleton sequence, with the occlusion ratio uniformly randomized between 10% and 30%. For occluded joints, their coordinates are set to 0. The occlusion region is selected based on a uniform distribution, ensuring that each joint has the same probability of being occluded, thus avoiding biased occlusion of specific joints or regions. The added noise is Gaussian noise with a mean of 0 and a standard deviation uniformly randomized between 0.1 and 0.3.

[0063] S4: constructing a neural network model for action recognition, then inputting the training set data into the neural network model in batches, and training the neural network model until a preset number of rounds is reached;

[0064] Specifically, after each batch of training is completed, check whether all batches of data have completed the prediction operation. If all batches of data have been predicted, the current round of training ends, the epoch number is increased by 1 and the next round of training begins. Otherwise, the model is continuously trained using the next batch of data. If the epoch number reaches the preset number of rounds, the training process is terminated.

[0065] Preferably, ST-GCN is selected as the neural network model for action recognition. If the model is found to perform poorly in a specific dataset or application scenario during the experiment, it can be replaced by other neural networks suitable for skeletal action recognition, such as AA-GCN, CTR-GCN, and HD-GCN, according to the actual situation.

[0066] The training set data is fed into the ST-GCN network in batches of batch_size = 32 (adjustable based on hardware resources and data size). Before input, the data is preprocessed, including adjusting the dimensionality of the skeleton sequence to the required format for network input. For each batch of data, the network's forward propagation calculates the model's prediction for each sample action category. The prediction and true label for each sample are recorded for subsequent calculation of the sample's overall difficulty level.

[0067] The process of training the model using any batch B of data is as follows:

[0068] S41: Calculating the comprehensive difficulty of the sample using prior knowledge and a posteriori knowledge, wherein the a posteriori knowledge is the model prediction accuracy, and the prior knowledge is the average confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm;

[0069] S411: Calculate the weight of the posterior knowledge using the number of rounds of the current epoch and the total number of rounds of epochs required for complete training. The weight of the posterior knowledge is positively correlated with the number of rounds of the current epoch. The calculation formula of the weight of the posterior knowledge is as follows:

[0070]

[0071] wherein, a is the weight of the posterior knowledge, epoch cur represents the round number of the current epoch, epoch total represents the total number of rounds of epochs required for training;

[0072] Since the posterior knowledge is less reliable at the initial stage of model training, a is positively correlated with the round number of the current epoch, that is, the weight of the posterior knowledge gradually increases as the training proceeds.

[0073] S412: calculating the weight of the prior knowledge using the weight of the posterior knowledge, wherein the calculation formula of the weight of the prior knowledge is as follows:

[0074] β = 1 - a;

[0075] wherein, β is the weight of the prior knowledge, which is set to be negatively correlated with epoch, gradually changing from 1 at the beginning to 0.5, because the model is weak in the early stage of training, so the quality of the reference sample, that is, the prior, is mainly referred to, and the prior and the posterior are referred to at the later stage.

[0076] S413: calculating the comprehensive difficulty of the sample using the weight of the posterior knowledge, the weight of the prior knowledge, the model prediction accuracy, and the average value of the confidence of all joints in the bone sample estimated by the pose estimation algorithm, wherein the calculation formula of the comprehensive difficulty of the sample is as follows:

[0077] s = a x P t + β x Q conf ;

[0078] wherein, s represents the comprehensive difficulty of the sample, the greater the value, the simpler the sample, a is the weight of the posterior knowledge, P t is the probability of the model prediction being correct, that is, the posterior, β is the weight of the prior knowledge, Q conf is the average value of the confidence of all joints in the bone sample estimated by the pose estimation algorithm, wherein the calculation formula of Q conf is as follows:

[0079]

[0080] wherein, q tvm represents the confidence of the t-th frame, the v-th joint, and the m-th person, T represents the total number of frames of the sample, V represents the total number of joints of the sample, and M represents the total number of people of the sample.

[0081] S42: Calculate the learning center of gravity at the current stage, and use the learning center of gravity and the comprehensive difficulty of the sample to calculate the sample loss weight. Then, use the sample loss weight to weight the cross entropy loss function to obtain the final loss;

[0082] S421: Calculate the learning center of gravity of the current epoch using the relationship between the current epoch number and the number of epochs required for complete training and manually set hyperparameters. The calculation formula for the learning center of gravity is as follows:

[0083]

[0084] Among them, k represents the learning center, A and B are hyperparameters used to set the size of the initial and final values, and epoch cur Represents the current epoch round number, epoch total Represents the total number of epochs required for training;

[0085] S422: Use the analytical expression of the chi-square distribution to calculate the loss weights of different samples, where the learning center of gravity is used as the degree of freedom of the chi-square distribution. The calculation formula for the loss weight of the sample is as follows:

[0086]

[0087] Among them, w represents the loss weight of the sample, k represents the learning center of gravity, s represents the comprehensive difficulty of the sample, the larger s is, the simpler the sample is, and Γ represents the Gamma function;

[0088] The above calculation method can moderately reduce the loss weight for samples that are simpler than the learning center of gravity, and significantly reduce the loss weight for samples that are more difficult than the learning center of gravity.

[0089] S423: Normalize the loss weights of the samples in batch B using a normalization method. The formula for normalizing the loss weights calculated for the samples in batch B to a certain interval is as follows:

[0090]

[0091] Among them, w norm is the normalized sample loss weight of the batch B, w min and w max It is a manually set hyperparameter, representing the maximum and minimum values ​​of the sample loss weight, usually 0 and 1, batch min and batch max They represent the maximum and minimum values ​​of the comprehensive difficulty of the samples in batch B, respectively, and w is the loss weight of the sample;

[0092] In step S423, the loss weight calculated for each batch of samples may be normalized to 0-1, or may be normalized to other reasonable intervals based on the quality distribution of the data set and the fitting ability of the model.

[0093] S424: Using the normalized sample loss weights of the batch B, weight the cross entropy loss function to obtain a final loss. The formula for weighting the cross entropy loss function is as follows:

[0094] Loss=w norm ×Loss CE ;

[0095] Among them, Loss is the final loss of the batch B, w norm is the normalized sample loss weight of batch B, Loss CE is the cross entropy loss function used to train classification neural networks.

[0096] S43: Update the neural network model using the final loss;

[0097] During the update process, the network parameters are updated using the stochastic gradient descent algorithm (or other optimization algorithms such as Adagrad and Adadelta, depending on the actual situation) at a set learning rate (e.g., 0.001). The learning rate can be adjusted based on the convergence during training. If the model converges too quickly, resulting in underfitting, the learning rate can be increased appropriately; if the model converges too slowly or oscillates, the learning rate can be reduced appropriately.

[0098] After that, the trained model can be used to perform action recognition on the image to be recognized.

[0099] Experiments have shown that the performance and robustness of the final model are significantly improved by using the above training method.

[0100] The multi-mass skeleton sequence action recognition method based on curriculum learning provided by the present invention fully utilizes the idea of ​​curriculum learning and learns the multi-mass skeleton sequences in sequence from easy to difficult.

[0101] The concept of curriculum learning draws on the human learning process, gradually transitioning from simple to complex knowledge. For multi-quality skeletal sequences, this learning sequence helps the model initially build a foundational understanding of basic movement patterns and skeletal structure. As learning progresses, the model gradually handles more complex and lower-quality skeletal sequences, thereby improving the model's learning efficiency and generalization capabilities.

[0102] In the evaluation of sample difficulty, the method uses different weights of prior and posterior knowledge at different stages to evaluate the comprehensive difficulty of the sample. Among them, the prior knowledge specifically covers multiple aspects of bone quality, such as the accuracy of the joint and the integrity of the bone structure. The posterior knowledge reflects the model prediction accuracy through the validation set accuracy and the stage accuracy in the training process.

[0103] In terms of training scheduling, the application adopts the idea of learning center, so that the training model has different learning centers at different stages. The learning center here gradually transitions from simple samples in the early stage to difficult samples, and its starting and ending positions are determined by reasonably setting hyperparameters. For samples simpler than the learning center, the loss weight is appropriately reduced to avoid the model paying too much attention to these relatively easy samples, so as to allocate more learning resources to more challenging samples. For samples more difficult than the learning center, the loss weight is greatly reduced to prevent the model from falling into a local optimal solution due to excessive pursuit of learning on these extremely difficult samples, affecting the overall learning efficiency.

[0104] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the application is indicated by the following claims.

[0105] It should be understood that the application is not limited to the details of the foregoing description, and that modifications and variations can be made without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

Claims

1. A multi-mass skeleton sequence action recognition method based on curriculum learning, characterized by: include: S1: Collect a large number of RGB videos containing human actions and with action category labels, and divide the collected videos into training and test sets; S2: Extract the coordinate information of the human joint points in each frame of the video and generate a continuous bone sequence according to the bone structure connection method; S3: performing data enhancement on the skeleton sequence to obtain a multi-quality skeleton sequence; S4: constructing a neural network model for action recognition, then inputting the training set data into the neural network model in batches, and training the neural network model until a preset number of rounds is reached; The process of training the model using any batch B of data is as follows: S41: Calculate the comprehensive difficulty of the sample using prior knowledge and a posteriori knowledge, where the a posteriori knowledge is the model prediction accuracy and the prior knowledge is the average confidence of all relevant nodes in the skeleton sample estimated using the posture estimation algorithm. The calculation formula for the comprehensive difficulty of the sample is as follows: s=α×P t +β×Q conf ; In the formula, s represents the comprehensive difficulty of the sample. The larger the value, the simpler the sample. α is the weight of the posterior knowledge. P t is the probability that the model predicts correctly, β is the weight of prior knowledge, Q conf It is the average confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm. As the number of training rounds increases, the weight of prior knowledge decreases, and the weight of posterior knowledge increases. S42: Calculate the learning center of gravity at the current stage, and use the learning center of gravity and the comprehensive difficulty of the sample to calculate the sample loss weight. Then, use the sample loss weight to weight the cross entropy loss function to obtain the final loss; Among them, as the number of training rounds increases, the learning focus gradually shifts from simple samples to difficult samples; The loss weights of different samples are calculated using the analytical expression of the chi-square distribution, where the learning center of gravity is used as the degree of freedom of the chi-square distribution. The loss weight of the sample is calculated as follows: Among them, w represents the loss weight of the sample, k represents the learning center of gravity, s represents the comprehensive difficulty of the sample, the larger the s, the simpler the sample, and Γ represents the Gamma function; S43: Update the neural network model using the final loss.

2. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: In S3, the specific steps of performing data enhancement on the skeleton sequence are as follows: performing a random occlusion operation on the skeleton sequence and adding random noise to the coordinates of the joint points in the skeleton sequence, wherein each occlusion randomly selects some joint points in the skeleton sequence, and the occlusion ratio is uniformly randomly taken between 10% and 30%. For the occluded joint points, their coordinate values ​​are set to 0, and the selection of the occlusion area follows the uniform distribution principle. The added noise is Gaussian noise with a mean of 0 and a standard deviation uniformly randomly taken between 0.1 and 0.

3.

3. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: In S41, the calculation formula of the weight of the posterior knowledge is as follows: Where α is the weight of the posterior knowledge, epoch cur Indicates the number of rounds of the current epoch, epoch total Indicates the total number of epochs required for training; The calculation formula of the weight of the prior knowledge is as follows: β＝1-α； Among them, β is the weight of prior knowledge.

4. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: In S41, the average confidence value Q of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm is conf The calculation formula is as follows: Where q tvm Represents the confidence of the t-th frame, the v-th joint point, and the m-th person. T represents the total number of frames of the sample, V represents the total number of joints in the sample, and M represents the total number of people in the sample.

5. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: In S42, the calculation formula for learning the center of gravity is as follows: Among them, k represents the learning center, A and B are hyperparameters used to set the size of the initial and final values, and epoch cur Represents the current epoch round number, epoch total Represents the total number of epochs required for training.

6. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: S42 further includes the following steps: The loss weights of the samples in batch B are normalized using a normalization method. The formula for normalizing the loss weights of the samples in batch B to a certain interval is as follows: Among them, w norm is the normalized sample loss weight of the batch B, w min and w max It is a manually set hyperparameter, representing the maximum and minimum values ​​of the sample loss weight, batch min and batch max They represent the maximum and minimum values ​​of the comprehensive difficulty of the samples in batch B, respectively, and w is the loss weight of the sample.

7. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 6 is characterized in that: In S42, the cross entropy loss function is weighted using the normalized loss weight of the batch B samples to obtain the final loss, wherein the formula for weighting the cross entropy loss function is as follows: Loss=w norm ×Loss CE ; Among them, Loss is the final loss of the batch B, w norm is the normalized sample loss weight of batch B, Loss CE is the cross entropy loss function used to train classification neural networks.

Images