Multi-quality skeleton sequence action recognition method based on course learning

Abstract

The invention discloses a multi-quality skeleton sequence action recognition method based on course learning, and the method comprises the steps: collecting a large number of videos which comprise human body actions and are provided with action category labels, and dividing the videos into a training set and a test set; extracting coordinate information of human body joint points of each frame of image in the video, and generating a continuous skeleton sequence; performing data enhancement on the skeleton sequence; according to the method, a neural network model for action recognition is constructed, then data of a training set is input into the neural network model in batches, the neural network model is trained until a preset round number is reached, in the process of training the model, prior knowledge and posterior knowledge are dynamically adjusted to evaluate sample difficulty, and in addition, the training efficiency is improved. The concept of learning the center of gravity is introduced, and the learning center of gravity gradually transits from an early simple sample to a difficult sample in different stages. The method provided by the invention can improve the recognition efficiency and the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition of computer vision, and in particular to a multi-quality skeleton sequence action recognition method based on curriculum learning. Background Art

[0002] The main goal of action recognition is to classify data containing one or more people performing specific actions. Currently, there are many common action recognition modalities, including visual modalities such as RGB video, skeleton sequence, depth data, infrared sequence, point cloud and event stream, as well as invisible modalities such as audio, speed, radar signal and WIFI signal. Each modality has its own unique advantages and disadvantages. Among them, skeleton sequence has outstanding advantages in action recognition. It has become one of the focuses of action recognition research with its simple and effective information, robustness to perspective shift and independence from background.

[0003] At present, research on high-quality skeleton sequences has achieved very good recognition performance. However, in practical application scenarios, the cost of obtaining high-quality skeleton sequences is often quite high. Due to technical and cost limitations, many devices can only use lightweight posture estimation networks to extract low-quality skeleton sequences from videos. When these low-quality skeleton sequences are input into a model trained with high-quality skeleton sequences, the accuracy is significantly reduced. Moreover, current research on low-quality skeletons is generally not well adapted to high-quality skeleton sequences, which poses a great challenge to practical applications.

[0004] Most of the previous methods were trained and evaluated based on relatively clean skeleton data. In recent years, more and more studies have begun to focus on how to deal with the problem of action recognition from noisy skeletons. These studies regard this problem as a challenge for skeleton denoising. First, linear or nonlinear transformations are used to filter skeleton noise, and then action recognition is performed. Among them, some methods use linear transformations for skeleton denoising based on biologically constrained skeleton structure priors. However, linear transformations that rely solely on human body priors may not be the best choice, and the effect of noise removal may not be strong enough. In order to overcome the shortcomings of linear transformations, PE-GCN proposed a nonlinear denoising transformation method based on autoencoders. However, this method is problematic in that it is only trained by forcing the mean square loss between the noise input and the reconstructed output, because this can easily cause the model to fall into a trivial solution (i.e., the same mapping). In addition, the unknown nature of the skeleton degradation process makes it challenging to find a universal denoising model and further promote action recognition.

[0005] The backbone GCN of the RA-GCN network consists of a series of ordered streams that are responsible for learning discriminative features from previously unactivated joints, and only the information of unactivated joints will be passed to the next stream. By activating some unimportant joints multiple times, the network is not only dependent on certain joints, thereby enhancing the robustness of the network. Other methods use predictive coding to train the model by maximizing the mutual information between normal skeletons and noisy skeletons. The knowledge distillation method pre-trains the teacher model with high-quality human skeleton sequence data, and then uses the pre-trained teacher model to guide the training of the student model with low-quality human skeleton sequence data. MaskCLR improves robustness by masking high-attention joints and forcing the network to learn the motion characteristics of the whole body.

[0006] Action recognition based on skeleton data has recently performed well on various types of datasets. However, due to the variety of quality of skeleton data extracted in practical applications, this may cause a sharp drop in the performance of action recognition. A simple solution is to make the network see as many different quality skeleton data as possible during training, so that the discriminative features of skeleton data of different qualities can be learned. Therefore, it is very necessary to find a way to make the existing methods better learn multi-quality skeleton data.

[0007] Therefore, proposing a new action recognition method for multi-mass human skeleton sequences has become an urgent problem to be solved. Summary of the invention

[0008] In view of this, the present invention provides a multi-mass skeleton sequence action recognition method based on curriculum learning to achieve accurate action recognition.

[0009] The technical solution provided by the present invention is specifically a multi-quality skeleton sequence action recognition method based on curriculum learning, comprising the following steps:

[0010] S1: Collect a large number of RGB videos containing human actions and with action category labels, and divide the collected videos into training sets and test sets;

[0011] S2: extract the coordinate information of the human joint points of each frame image in the video, and generate a continuous bone sequence according to the bone structure connection method;

[0012] S3: performing data enhancement on the skeleton sequence to obtain a multi-quality skeleton sequence;

[0013] S4: constructing a neural network model for action recognition, and then inputting the data of the training set into the neural network model in batches, and training the neural network model until a preset number of rounds is reached;

[0014] The process of training the model using any batch B of data is as follows:

[0015] S41: Calculate the comprehensive difficulty of the sample using prior knowledge and a posteriori knowledge, wherein the a posteriori knowledge is the model prediction accuracy, and the prior knowledge is the average value of the confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm, wherein the calculation formula of the comprehensive difficulty of the sample is as follows:

[0016] s=α×P t +β×Q conf ;

[0017] In the formula, s represents the comprehensive difficulty of the sample. The larger the value, the simpler the sample. α is the weight of the posterior knowledge. t is the probability that the model predicts correctly, β is the weight of prior knowledge, and Q conf It is the average value of the confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm. As the number of training rounds increases, the weight of prior knowledge decreases, and the weight of posterior knowledge increases.

[0018] S42: Calculate the learning center of gravity at the current stage, and use the learning center of gravity and the comprehensive difficulty of the sample to calculate the sample loss weight, and then use the sample loss weight to weight the cross entropy loss function to obtain the final loss;

[0019] Among them, as the number of training rounds increases, the focus of learning gradually shifts from simple samples to difficult samples;

[0020] The loss weights of different samples are calculated using the analytical expression of the chi-square distribution, where the learning center of gravity is used as the degree of freedom of the chi-square distribution. The loss weight of the sample is calculated as follows:

[0021]

[0022] Among them, w represents the loss weight of the sample, k represents the learning center of gravity, s represents the comprehensive difficulty of the sample, the larger s is, the simpler the sample is, and Γ represents the Gamma function;

[0023] S43: Update the neural network model using the final loss.

[0024] Preferably, in S3, the specific steps of performing data enhancement on the skeleton sequence are as follows: performing a random occlusion operation on the skeleton sequence and adding random noise to the coordinates of the joint points in the skeleton sequence, wherein each occlusion randomly selects some joint points in the skeleton sequence, and the occlusion ratio is uniformly randomly taken between 10% and 30%. For the occluded joint points, their coordinate values ​​are set to 0, and the selection of the occlusion area follows the principle of uniform distribution. The added noise is Gaussian noise with a mean of 0 and a standard deviation uniformly randomly taken between 0.1 and 0.3.

[0025] Further preferably, in S41, the calculation formula of the weight of the a posteriori knowledge is as follows:

[0026]

[0027] In the formula, α is the weight of the posterior knowledge, epoch cur Indicates the round number of the current epoch, epoch total Indicates the total number of epochs required for training;

[0028] The calculation formula of the weight of the prior knowledge is as follows:

[0029] β = 1-α;

[0030] Among them, β is the weight of prior knowledge.

[0031] Further preferably, in S41, the average value Q of the confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm is conf The calculation formula is as follows:

[0032]

[0033] In the formula, q tvm represents the confidence of the t-th frame, the v-th joint point, and the m-th person. T represents the total number of frames of the sample, V represents the total number of joints in the sample, and M represents the total number of people in the sample.

[0034] Further preferably, in S42, the calculation formula of the learning center of gravity is as follows:

[0035]

[0036] Among them, k represents the learning center, A and B are hyperparameters used to set the size of the initial and final values, and epoch cur Represents the current epoch round number, epoch total Represents the total number of epochs required for training.

[0037] Further preferably, S42 further includes the following steps:

[0038] The loss weights of the samples in the batch B are normalized using a normalization method, wherein the formula for normalizing the loss weights of the samples in the batch B to a certain interval is as follows:

[0039]

[0040] Among them, w norm is the normalized sample loss weight of the batch B, w min and w max is a manually set hyperparameter, representing the maximum and minimum values ​​of the sample loss weight, batch min and batch max They respectively represent the maximum and minimum values ​​of the comprehensive difficulty of the samples in batch B, and w is the loss weight of the samples.

[0041] Further preferably, in S42, the cross entropy loss function is weighted using the normalized loss weight of the batch B samples to obtain a final loss, wherein the formula for weighting the cross entropy loss function is as follows:

[0042] Loss = w norm ×Loss CE ;

[0043] Among them, Loss is the final loss of the batch B, w norm is the normalized sample loss weight of batch B, Loss CE is the cross entropy loss function used to train classification neural networks.

[0044] The multi-quality skeleton sequence action recognition method based on curriculum learning provided by the present invention has the following advantages:

[0045] 1. The present invention innovatively uses prior and a posteriori knowledge to evaluate bone samples simultaneously, and the weights of prior and a posteriori knowledge can be dynamically changed according to the training process. In the early stage of training, the model has limited understanding of data, and now the reliability of a posteriori knowledge (such as model prediction accuracy) is low, so the weight of prior knowledge (bone quality) is relatively high, mainly relying on bone quality to guide the model to learn relatively simple and quality guaranteed samples, avoiding the model from over-relying on unreliable a posteriori knowledge and producing wrong learning direction. Along with the advancement of training, the model gradually deepens the learning of data, the reliability of a posteriori knowledge is enhanced, and its weight increases accordingly, and the a priori knowledge weight gradually decreases. This dynamic adjustment mechanism ensures that in different training stages, sample evaluation can make full use of the advantages of two kinds of knowledge, avoiding the unreliability caused by simply relying on a posteriori knowledge in the early stage due to the insufficient learning of the model, and the limitation of a priori knowledge in the later stage due to being unable to reflect the actual learning effect of the model.

[0046] 2. The present invention proposes a novel and unique learning center of gravity theory. In the process of training the model, the learning center of gravity will gradually transition from the early simple samples to the difficult samples as the training stage changes, and its starting and ending positions are accurately determined by reasonably setting the hyperparameters. For samples that are simpler than the learning center of gravity, the loss weight is moderately reduced so that the model does not pay too much attention to these relatively easy-to-master samples during the learning process, thereby allocating more learning resources to more challenging samples. For samples that are more difficult than the learning center of gravity, the loss weight is greatly reduced to prevent the model from falling into a local optimal solution due to excessive pursuit of learning these extremely difficult samples, affecting the overall learning efficiency.

[0047] 3. The method proposed by the present invention is particularly suitable for the situation that the quality of bones is diverse and uniform, which is a great advantage in practical applications. When training the model, high-quality data can be used to make low-quality data in varying degrees by specific data enhancement techniques (such as random occlusion, adding random noise, etc., which can briefly explain the principle and effect of data enhancement here), and these data are added to training together. This method can allow the model to fully contact the skeleton sequence of various qualities during the training process, learn the discriminant features of different quality skeleton data, thereby effectively improving the robustness of the model to low-quality skeleton sequences.

[0048] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of a multi-mass skeleton sequence action recognition method based on curriculum learning is provided in an embodiment disclosed by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] In view of the problem that current technology cannot efficiently utilize sample information in multi-quality data processing, resulting in limited model performance, this implementation provides a multi-quality skeleton sequence action recognition method based on curriculum learning, see Figure 1 , the method comprises the following steps:

[0054] S1: Collect a large number of RGB videos containing human actions and with action category labels, and divide the collected videos into training sets and test sets;

[0055] Preferably, the training set and the test set are divided in a ratio of 7:3;

[0056] The division process uses a stratified random sampling method to ensure that the distribution ratio of each action category in the training set and the test set is similar, avoiding the impact of data bias on model training.

[0057] S2: extracting the coordinate information of the human joint points of each frame image in the video, and generating a continuous bone sequence according to a specific bone structure connection method (such as a commonly used 2D human bone structure model);

[0058] The coordinate information of the human body joints can be obtained by processing the RGB video obtained in S1 using a selected posture estimation method. The posture estimation method can be selected as a default posture estimation method from OpenPose, Pifpaf, HRNet, VITPose and other methods that can be used for 2D posture estimation according to the actual application scenario and the requirements for accuracy and speed.

[0059] In the process of generating a continuous skeleton sequence, the joint point coordinates are normalized so that their value range is between [0,1] to facilitate subsequent calculation and processing.

[0060] S3: performing data enhancement on the skeleton sequence to obtain a multi-quality skeleton sequence;

[0061] The specific steps are as follows: perform a random occlusion operation on the skeleton sequence and add random noise to the coordinates of the joint points in the skeleton sequence to simulate the measurement errors and noise interference that may occur in practical applications. Through the above-mentioned data enhancement method, a multi-quality skeleton sequence is obtained to increase the diversity and complexity of the data and improve the model's generalization ability for skeleton data of different qualities.

[0062] Each time the occlusion is performed, some joints in the skeleton sequence are randomly selected, and the occlusion ratio is uniformly randomly selected between 10% and 30%. For the occluded joints, their coordinate values ​​are set to 0. The selection of the occluded area follows the principle of uniform distribution, that is, the probability of each joint being occluded is the same, avoiding biased occlusion of specific joints or areas; the added noise is Gaussian noise with a mean of 0 and a standard deviation uniformly randomly selected between 0.1 and 0.3.

[0063] S4: constructing a neural network model for action recognition, and then inputting the data of the training set into the neural network model in batches, and training the neural network model until a preset number of rounds is reached;

[0064] Specifically, after each batch of training is completed, check whether all batches of data have completed the prediction operation. If all batches of data have been predicted, the current round of training ends, and the epoch number is increased by 1 to enter the next round of training. Otherwise, the next batch of data is used to continue training the model. If the epoch number reaches the preset number of rounds, the training process is terminated.

[0065] Preferably, ST-GCN is selected as the neural network model for action recognition. If it is found during the experiment that the model does not perform well in a specific data set or application scenario, it can be replaced by other neural networks suitable for skeletal action recognition, such as AA-GCN, CTR-GCN, HD-GCN, etc., according to the actual situation;

[0066] The training set data is input into the ST-GCN network in batches according to the size of batch_size=32 (which can be adjusted according to hardware resources and data scale). Before input, the data is preprocessed, including adjusting the dimension of the skeleton sequence to the format required by the network input. For each batch of data, the model's prediction results for each sample action category are calculated through the network's forward propagation. At the same time, the prediction results and true labels of each sample are recorded for the subsequent calculation of the comprehensive difficulty of the sample.

[0067] The process of training the model using any batch B of data is as follows:

[0068] S41: Calculate the comprehensive difficulty of the sample using prior knowledge and a posteriori knowledge, wherein the a posteriori knowledge is the model prediction accuracy, and the prior knowledge is the average value of the confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm;

[0069] S411: Calculate the weight of the a posteriori knowledge using the number of rounds of the current epoch and the total number of rounds of epochs required for complete training, wherein the weight of the a posteriori knowledge is positively correlated with the number of rounds of the current epoch. The calculation formula of the weight of the a posteriori knowledge is as follows:

[0070]

[0071] In the formula, α is the weight of the posterior knowledge, epoch cur Indicates the round number of the current epoch, epoch total Indicates the total number of epochs required for training;

[0072] Because the reliability of posterior knowledge is poor in the early stage of model training, α is positively correlated with the current epoch number, that is, as the training progresses, the weight of posterior knowledge gradually increases.

[0073] S412: Calculate the weight of the prior knowledge using the weight of the a posteriori knowledge, wherein the calculation formula of the weight of the prior knowledge is as follows:

[0074] β = 1-α;

[0075] Among them, β is the weight of prior knowledge, which is set to be negatively correlated with the epoch, gradually changing from 1 at the beginning to 0.5. Because the model's ability is weak in the early stage of training, the quality of the main reference sample is the prior, while in the later stage, both the prior and the posteriori are referenced.

[0076] S413: Calculate the comprehensive difficulty of the sample using the weight of the a priori knowledge, the weight of the a priori knowledge, the model prediction accuracy, and the average value of the confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm, wherein the calculation formula of the comprehensive difficulty of the sample is as follows:

[0077] s=α×P t +β×Q conf ;

[0078] In the formula, s represents the comprehensive difficulty of the sample. The larger the value, the simpler the sample. α is the weight of the posterior knowledge. t is the probability that the model predicts correctly, that is, the posterior, β is the weight of the prior knowledge, Q conf It is the average confidence of all nodes in the skeleton sample estimated by the posture estimation algorithm, where Q conf The calculation formula is as follows:

[0079]

[0080] In the formula, q tvm represents the confidence of the t-th frame, the v-th joint point, and the m-th person. T represents the total number of frames of the sample, V represents the total number of joints in the sample, and M represents the total number of people in the sample.

[0081] S42: Calculate the learning center of gravity at the current stage, and use the learning center of gravity and the comprehensive difficulty of the sample to calculate the sample loss weight, and then use the sample loss weight to weight the cross entropy loss function to obtain the final loss;

[0082] S421: Calculate the learning center of gravity of the current epoch using the relationship between the current epoch number and the number of epochs required for complete training and manually set hyperparameters, where the calculation formula for the learning center of gravity is as follows:

[0083]

[0084] Among them, k represents the learning center, A and B are hyperparameters used to set the size of the initial and final values, and epoch cur Represents the current epoch round number, epoch total Represents the total number of epochs required for training;

[0085] S422: Use the analytical expression of the chi-square distribution to calculate the loss weights of different samples, where the learning center of gravity is used as the degree of freedom of the chi-square distribution. The calculation formula for the loss weight of the sample is as follows:

[0086]

[0087] Among them, w represents the loss weight of the sample, k represents the learning center of gravity, s represents the comprehensive difficulty of the sample, the larger s is, the simpler the sample is, and Γ represents the Gamma function;

[0088] Among them, the above calculation method can achieve moderate reduction of loss weight for samples that are simpler than the learning center of gravity, and greatly reduce loss weight for samples that are more difficult than the learning center of gravity;

[0089] S423: Normalize the loss weights of the samples in the batch B using a normalization method, wherein the formula for normalizing the loss weights calculated for the samples in the batch B to a certain interval is as follows:

[0090]

[0091] Among them, w norm is the normalized sample loss weight of the batch B, w min and w max It is a manually set hyperparameter, representing the maximum and minimum values ​​of the sample loss weight, usually 0 and 1, batch min and batch max They represent the maximum and minimum values ​​of the comprehensive difficulty of the samples in batch B, respectively, and w is the loss weight of the samples;

[0092] In step S423, the loss weight calculated for each batch of samples may be normalized to 0-1, or may be normalized to other reasonable intervals according to the quality distribution of the data set and the fitting ability of the model.

[0093] S424: Use the normalized sample loss weights of the batch B to weight the cross entropy loss function to obtain a final loss, wherein the formula for weighting the cross entropy loss function is as follows:

[0094] Loss = w norm ×Loss CE ;

[0095] Among them, Loss is the final loss of the batch B, w norm is the normalized sample loss weight of batch B, Loss CE is the cross entropy loss function used to train classification neural networks.

[0096] S43: Update the neural network model using the final loss;

[0097] During the update process, the stochastic gradient descent algorithm (or other optimization algorithms such as Adagrad, Adadelta, etc., depending on the actual situation) is used to update the network parameters according to the set learning rate (for example, 0.001). The learning rate can be adjusted according to the convergence during the training process. If the model converges too quickly and causes underfitting, the learning rate can be appropriately increased; if the model converges too slowly or oscillates, the learning rate can be appropriately reduced.

[0098] After that, the trained model can be used to perform action recognition on the image to be recognized.

[0099] Experiments have shown that the performance and robustness of the final model are significantly improved by using the above training method.

[0100] The multi-mass skeleton sequence action recognition method based on curriculum learning provided by the present invention makes full use of the idea of ​​curriculum learning and sequentially learns the multi-mass skeleton sequences in the order of easy first and difficult later.

[0101] The idea of ​​curriculum learning draws on the human learning process, which gradually transitions from simple knowledge to complex knowledge learning mode. For multi-quality bone sequences, this learning order helps the model build a basic understanding of basic action patterns and bone structure in the initial stage. As learning progresses, the model gradually processes more complex and lower-quality bone sequences, thereby improving the model's learning efficiency and generalization ability.

[0102] When evaluating the difficulty of a sample, this method uses prior and posterior knowledge with different weights at different times to evaluate the comprehensive difficulty of the sample. Among them, the prior knowledge specifically covers multiple aspects of bone quality, such as the accuracy of joint points and the integrity of bone structure. The posterior knowledge reflects the model prediction accuracy through the accuracy of the validation set and the stage accuracy during the training process.

[0103] In terms of training scheduling, the present invention adopts the idea of ​​learning center of gravity so that the training model has different learning centers of gravity at different stages. The learning center of gravity here gradually transitions from simple samples in the early stage to difficult samples, and its starting and ending positions are determined by reasonably setting hyperparameters. For samples that are simpler than the learning center of gravity, the loss weight is moderately reduced to avoid the model from paying too much attention to these relatively easy samples, thereby allocating more learning resources to more challenging samples. For samples that are more difficult than the learning center of gravity, the loss weight is greatly reduced to prevent the model from falling into a local optimal solution due to excessive pursuit of learning these extremely difficult samples, affecting the overall learning efficiency.

[0104] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0105] It should be understood that the present invention is not limited to what has been described above and that various modifications and changes may be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A multi-quality skeleton sequence action recognition method based on curriculum learning, characterized in that: include: S1: Collect a large number of RGB videos containing human actions and with action category labels, and divide the collected videos into training sets and test sets; S2: extract the coordinate information of the human joint points of each frame image in the video, and generate a continuous bone sequence according to the bone structure connection method; S3: performing data enhancement on the skeleton sequence to obtain a multi-quality skeleton sequence; S4: constructing a neural network model for action recognition, and then inputting the data of the training set into the neural network model in batches, and training the neural network model until a preset number of rounds is reached; The process of training the model using any batch B of data is as follows: S41: Calculate the comprehensive difficulty of the sample using prior knowledge and a posteriori knowledge, wherein the a posteriori knowledge is the model prediction accuracy, and the prior knowledge is the average value of the confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm, wherein the calculation formula of the comprehensive difficulty of the sample is as follows: s=α×P t +β×Q conf ; In the formula, s represents the comprehensive difficulty of the sample. The larger the value, the simpler the sample. α is the weight of the posterior knowledge. t is the probability that the model predicts correctly, β is the weight of prior knowledge, and Q conf It is the average value of the confidence of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm. As the number of training rounds increases, the weight of prior knowledge decreases, and the weight of posterior knowledge increases. S42: Calculate the learning center of gravity at the current stage, and use the learning center of gravity and the comprehensive difficulty of the sample to calculate the sample loss weight, and then use the sample loss weight to weight the cross entropy loss function to obtain the final loss; Among them, as the number of training rounds increases, the learning focus gradually shifts from simple samples to difficult samples; The loss weights of different samples are calculated using the analytical expression of the chi-square distribution, where the learning center of gravity is used as the degree of freedom of the chi-square distribution. The loss weight of the sample is calculated as follows: Among them, w represents the loss weight of the sample, k represents the learning center of gravity, s represents the comprehensive difficulty of the sample, the larger s is, the simpler the sample is, and Γ represents the Gamma function; S43: Update the neural network model using the final loss.

2. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: In S3, the specific steps of performing data enhancement on the skeleton sequence are as follows: performing a random occlusion operation on the skeleton sequence and adding random noise to the coordinates of the joint points in the skeleton sequence, wherein each occlusion randomly selects some joint points in the skeleton sequence, and the occlusion ratio is uniformly randomly taken between 10% and 30%. For the occluded joint points, their coordinate values ​​are set to 0, and the selection of the occlusion area follows the principle of uniform distribution. The added noise is Gaussian noise with a mean of 0 and a standard deviation uniformly randomly taken between 0.1 and 0.

3.

3. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: In S41, the calculation formula of the weight of the a posteriori knowledge is as follows: In the formula, α is the weight of the posterior knowledge, epoch cur Indicates the round number of the current epoch, epoch total Indicates the total number of epochs required for training; The calculation formula of the weight of the prior knowledge is as follows: β＝1-α； Among them, β is the weight of prior knowledge.

4. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: In S41, the average confidence value Q of all relevant nodes in the skeleton sample estimated by the posture estimation algorithm is conf The calculation formula is as follows: In the formula, q tvm represents the confidence of the t-th frame, the v-th joint point, and the m-th person. T represents the total number of frames of the sample, V represents the total number of joints in the sample, and M represents the total number of people in the sample.

5. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: In S42, the calculation formula for learning the center of gravity is as follows: Among them, k represents the learning center, A and B are hyperparameters used to set the size of the initial and final values, and epoch cur Represents the current epoch round number, epoch total Represents the total number of epochs required for training.

6. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 1 is characterized in that: S42 further includes the following steps: The loss weights of the samples in the batch B are normalized using a normalization method, wherein the formula for normalizing the loss weights of the samples in the batch B to a certain interval is as follows: Among them, w norm is the normalized sample loss weight of the batch B, w min and w max is a manually set hyperparameter, representing the maximum and minimum values ​​of the sample loss weight, batch min and batch max They respectively represent the maximum and minimum values ​​of the comprehensive difficulty of the samples in batch B, and w is the loss weight of the samples.

7. The multi-mass skeleton sequence action recognition method based on curriculum learning according to claim 6 is characterized in that: In S42, the cross entropy loss function is weighted using the normalized loss weight of the batch B samples to obtain a final loss, wherein the formula for weighting the cross entropy loss function is as follows: Loss=w norm ×Loss CE ; Among them, Loss is the final loss of the batch B, w norm is the normalized sample loss weight of batch B, Loss CE is the cross entropy loss function used to train classification neural networks.

Images