Human skeleton generation method for behavior recognition

Abstract

The application discloses a human body skeleton generation method for behavior recognition, and steps include: 1) collecting a data set of related human body skeleton actions; 2) pre-processing the data set, selecting action category samples that need to be generated and complete action samples that need to be referenced, and dividing a training set and a test set; 3) building a skeleton style migration characteristic generation network, the generation network is composed of an encoder, a decoder and a discriminator, wherein the encoder and the decoder together constitute a generator, and the encoder is further subdivided into a content encoder and a style encoder; 4) using the training set to sequentially train the content encoder, the style encoder, the decoder and the discriminator; 5) the generated new sample is used to participate in the training of a behavior recognition network, the generalization ability of the behavior recognition network is improved, and the accuracy of the test set on the action is improved. The method has better accuracy of human body skeleton generation results and better recognition effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human skeleton posture generation, and relates to a human skeleton generation method for behavior recognition. BACKGROUND

[0002] Behavior recognition technology is one of the joint technologies that enable computers to understand human behavior, and has wide application prospects in smart care, intelligent monitoring, video retrieval, and intelligent home, etc. Human behavior recognition deep learning methods can be roughly divided into recognition methods based on RGB video signals and methods based on human skeleton models. The method based on RGB video signal refers to inputting a time sequence of image signals to the network, and its advantage lies in that the RGB image itself contains rich background information, which is conducive to the recognition of human interaction and specific scene, but the background information of most actions without objects will interfere with them. The other method, the method based on human skeleton model, refers to estimating the coordinates of the human skeleton joints by using human posture estimation method, and then using the coordinates of the human joints to represent human behavior and action for recognition and classification. The pure joint coordinates remove the redundant information in the background, so that the network pays more attention to the change of human action itself. Therefore, although this kind of method has simple and direct input signal, it also occupies a place in the field of behavior recognition.

[0003] No matter what kind of recognition strategy, as long as it is a deep learning related method, it is necessary to rely on the sufficiency of training samples. At present, the publicly recognized behavior recognition data sets in the academic field are UCF101, NTU-RGBD60, NTU-RGBD120, Human3.6, and Kinetics, etc. The making of these data sets not only needs to make corresponding annotations on image video data, but also needs to obtain the joint coordinate information of the human body in each frame of image. This process consumes a lot of manpower and material resources. Secondly, the actions that need to be recognized in the actual behavior recognition application scene are generally not included in the public data set, such as falling down, fighting, climbing over the wall, etc. Therefore, this problem is a difficulty and a valuable research direction in the field of behavior recognition research.

[0004] With the continuous development of unsupervised learning in deep learning, the method of generating a large number of samples by using unsupervised learning can solve the problem of sample shortage to a certain extent. The mainstream generation methods include variational autoencoder (VAE), generative adversarial network (GAN) and diffusion model (Diffusion Model). Due to the high complexity of image data, the network itself is difficult to learn the data distribution, and it is more difficult to generate continuous video signals. The existing video generation algorithm is single in the video environment, and the generated effect is poor, which will appear the situation of incoherent between frames and collapse of some frame modes, so it cannot be applied to actual behavior recognition. However, generating human skeleton and human skin is also a new method in recent years. Most of these methods are based on the experience of generating images, and the generated skeleton is used for expanding the dataset samples, animation industry and the like. SUMMARY

[0005] The purpose of the present application is to provide a human skeleton generation method for behavior recognition, which solves the problem of high production cost of public behavior recognition dataset and difficulty of public data category to meet the actual behavior recognition task in the prior art.

[0006] The technical scheme adopted by the present application is a human skeleton generation method for behavior recognition, which is implemented according to the following steps:

[0007] Step 1, collect the dataset of related human skeleton actions;

[0008] Step 2, pre-process the dataset obtained in step 1, select the action category samples to be generated and the complete action samples to be used for reference style, and divide the training set and the test set;

[0009] Step 3, build a skeleton style transfer feature generation network,

[0010] The generation network is composed of an encoder, a decoder and a discriminator, wherein the encoder and the decoder together constitute a generator, and the encoder is further divided into a content encoder and a style encoder;

[0011] Step 4, train the content encoder, the style encoder, the decoder and the discriminator in turn by using the training set;

[0012] Step 5, participate in the training of the behavior recognition network with the generated new samples, improve the generalization ability of the behavior recognition network, and improve the accuracy of the action on the test set.

[0013] The present application has the following beneficial effects:

[0014] 1) The method of the present application is based on the idea of style transfer in images as a whole, and the network is composed of an autoencoder network and an adversarial network, wherein the encoding and decoding network serves as a generator in the adversarial network and can completely complete the task of sample generation, and the discriminator in the adversarial network assists in training; the encoding and decoding network is composed of a skeletal content encoder and a skeletal style encoder and a decoder, which can encode the skeletal data into corresponding content vectors and skeletal vectors, and better complete the generation task of skeletal data under the condition of less samples while realizing the decoupling of feature content and style.

[0015] 2) The method of the present application can generate a large number of skeletal action samples by borrowing the action features of complete data, which can significantly reduce the cost of manually labeling skeletal data; using the generated skeletal samples for data augmentation can effectively improve the generalization ability of the current mainstream human behavior recognition network model based on skeletal data, further reducing the occurrence of overfitting in the training process; training the human recognition network through the generated samples can significantly improve the recognition rate of the network for this type of action. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is the principle block diagram in the method of the present application;

[0017] Figure 2 is the network block diagram of skeletal style transfer in the method of the present application;

[0018] Figure 3 is the content encoder block diagram in the method of the present application;

[0019] Figure 4 is the style encoder block diagram in the method of the present application;

[0020] Figure 5 is the decoder block diagram in the method of the present application;

[0021] Figure 6a is the drinking action experiment result display of the embodiment of the present application;

[0022] Figure 6b is the jumping action experiment result display of the embodiment of the present application. DETAILED DESCRIPTION

[0023] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0024] Reference Figure 1The technical route of the method is that, aiming at the problem that the action to be recognized in the actual application scene is difficult to obtain, such as wall climbing, fighting, technical foul, etc., research ideas are developed. First, if a small amount of action samples are directly generated by using a generation network, due to the too small number of training samples, the generation network is often over-fitted, the generation effect is poor, and the diversity is low, and other adverse results. Therefore, the present application attempts to make the generation network fully learn the distribution of a relatively complete action sample, and reasonably and feasibly migrate the diversity to the data set to be expanded, so as to solve the problem of over-fitting caused by too few training samples, and fully utilize the completeness of the action sample distribution of the existing data set, to a certain extent, the purpose of expanding the sample can be achieved; then, the generated sample is actually participated in the training of the behavior recognition network, and the recognition rate of the test set of the behavior recognition network and the subjective evaluation of the generated sample are tested from multiple aspects to show whether the generated sample effectively solves the problem.

[0025] Reference Figure 2 The human body skeleton data generation method for behavior recognition according to the present application is implemented according to the following steps:

[0026] Step 1, collect the data set of related human body skeleton action,

[0027] Select an existing human body behavior recognition data set, for example, UCF101, NTURGBD60, NTURGBD60 data set, these human body behavior recognition data sets not only contain human body action RGB video, but also contain corresponding human body skeleton coordinate data, which can be used for the task of generating skeleton; or, a Kinetics sensor self-made data set is used, and the joint point coordinates of the human body wearing the device are obtained by using the sensor,

[0028] Then, the collected data is classified and labeled with action categories for training.

[0029] Step 2, pre-process the data set obtained in step 1, select the action category sample to be generated and the complete action sample to be referenced, and divide the training set and the test set,

[0030] The specific process is,

[0031] First, select the action category data to be generated and the complete distribution action data to be referenced;

[0032] Second, the samples with large errors are manually excluded, and the data is normalized, for each group of samples, the coordinates of all joint points are divided by the maximum value, so that the size of the joint point coordinates is in the range of [0, 1];

[0033] Then, the bone relative coordinates are converted into absolute coordinate values, i.e. taking a certain joint of the human body as a coordinate origin, performing a translation operation, and subtracting the dimension coordinates of the reference joint from the dimension coordinates of all joints;

[0034] Finally, the training set and the test set are divided.

[0035] Step 3, building a generation network with a bone style transfer feature,

[0036] The generation network is composed of an encoder, a decoder and a discriminator, wherein the encoder and the decoder together constitute a generator, and the encoder is further divided into a content encoder and a style encoder,

[0037] The speed of the bone data participating in the training of the generation network in the NTURGBD60 data set is calculated, the speed is obtained by subtracting the coordinate points of adjacent frames, see formula (1), the joint coordinate is input into the content encoder, and the speed value is input into the style encoder, and then there is:

[0038] v t+1 =x t+1 -x t (1)

[0039] Wherein, x is the joint coordinate of the human body, v is the speed value calculated at this moment, and t is the current moment;

[0040] Referring to Figure 3 , the structural principle of the content encoder, the input data here is the bone data (T, p x , p y , p z ) of the human body, T represents the Tth frame in the time dimension, and the remaining three numerical values represent the coordinate values in the X, Y and Z directions respectively; The input coordinate values are obtained through 3 times of 1D convolution with a step of 2 and a BN layer in series to obtain the final bone vector. Since the content encoder uses 1D convolution, the bone vector features extracted by the network are more focused on the position information of the joint space, so the obtained features can represent the content of the action; In addition, the use of the BN normalization layer (Batch Normalization) can speed up the convergence of the network, is not easy to diverge, and can improve the accuracy and speed of the training.

[0041] Referring to Figure 4 , the structural principle of the style encoder, the input data here is the bone data (T, v x , v y , v z), T represents the Tth frame in the time dimension, and the remaining three values represent the velocity values in the X, Y, Z directions, respectively, which are calculated from the original data by formula (1); the input velocity values pass through 3 times of 1D convolution and BN layer modules in series with a step size of 1, and then pass through an LSTM layer to obtain the final velocity vector. Since the style encoder adds an LSTM module on the basis of the original 1D convolution and BN layer, the extracted features contain time sequence information of data changes, so the obtained features can represent the style of the action. The specific LSTM calculation value is shown in formula (2):

[0042] h t = σ(W xh x t + W hh h t-1 + b) (2)

[0043] wherein W xh , W hh represent network parameters to be trained, x t represents a feature value of a current node, h t-1 represents a feature value accumulated at a previous moment, b represents a bias feature value, and σ() represents a nonlinear activation function.

[0044] Referring to Figure 5 , the structure principle of the decoder, the content vector obtained by the foregoing two encoders is input together with the velocity vector, the content vector is taken as the main body, and the content vector passes through 3 layers of 1D convolution, SPAdaIN residual blocks, an LSTM network layer and a layer of 1D convolution to obtain the skeleton data Y; the velocity vector passes through 3 layers of Linear fully connected layers, and then SPAdaIN operations are performed on the features at each stage of the content vector, so as to correct the features to obtain the features of the finally generated sample. The specific SPAdaIN operation calculation is shown in formula (3):

[0045]

[0046] wherein x and v represent the input content vector and style vector respectively, and μ and σ represent the mean and standard deviation of the corresponding features respectively. The physical meaning of the formula (3) is to subtract the mean divided by the standard deviation of itself, then multiply by the standard deviation of the target feature, and then add the mean of the target feature. In essence, it is a process of self-style normalization and adding target-style reverse normalization. A large number of papers and experiments have proved that the AdaIN module can complete the style transfer operation.

[0047] The structure of the discriminator is composed of a plurality of traditional 1D convolutions in series. The discriminator plays an auxiliary training role. The number of all 1D convolutions in series should not be too large, and needs to be dynamically adjusted according to the complexity of the actual action.

[0048] From then on, in the training process, the finally generated skeleton data is fed into the discriminator together with the skeleton data participating in the training of the generation network to obtain a complete generation network.

[0049] Step 4, using the training set to train the content encoder, the style encoder, the decoder and the discriminator in turn, the specific process is:

[0050] In the training process, the reconstruction loss of L1 distance and the W distance (i.e. Wasserstein distance) are used as the loss function,

[0051] 4.1) initialization of training parameters,

[0052] The training parameters are set as follows: epoch is the number of times of training all data, which is set to an integer between 500 and 800; batch_size is the number of samples in each batch training, which is set to one of {8, 16, 32, 64}; learning_rate is the learning rate, and the initial learning rate is 0.0001;

[0053] 4.2) training of the generation network using the coding and decoding reconstruction loss,

[0054] The input skeleton data of a small amount of action is calculated according to formula (1) to obtain the velocity data, the skeleton data of a small amount of action is input into the content encoder to obtain the content vector, the corresponding velocity data is input into the style encoder to obtain the style vector, and the two content vectors and style vectors are input into the decoder. At this time, the input is the original content and style, and therefore the output of the decoder corresponds to the original skeleton data. The loss function uses the reconstruction loss of L1 distance, as shown in formula (4):

[0055] L re = ||Y-X|| (4)

[0056] 4.3) training of the generation network using the adversarial loss,

[0057] As mentioned before, the generation network is not enough to have the ability to restore the original sample, and also needs to have the ability to generate. At this time, the reference complete skeleton data needs to be operated according to step 4.2) again to obtain the reference content vector and the reference style vector. The reference style vector is separated from the original content vector and input into the decoding end to obtain the new skeleton data. At this time, the generation network has the ability to generate samples. Since there is no standard for comparison for the generated samples, the adversarial loss is used again, the generated samples and the original samples are input into the discriminator to judge the true and false scores, the adversarial training of the generation network, and the W distance loss function is shown in formula (5) and formula (6):

[0058] L G = -D(G(x,v)) (5)

[0059] L D =D(G(x,v))-D(y)+l(||gradD(x,v)||2-1) 2 (6)

[0060] Wherein, G(x,v) represents the generated new sample, D() represents the score of the input data through the discriminator, and lambda is the coefficient corresponding to the W distance as a hyperparameter dynamically adjusted.

[0061] Step 5, the generated new sample G(x,v) is involved in the training of the behavior recognition network, the generalization ability of the behavior recognition network is improved, and the accuracy of the action on the test set is improved.

[0062] This step 5 is a step of test evaluation, the behavior recognition network selects a public STGCN network or a PBGCN network, the generated new sample G(x,v) is involved in the training, and the accuracy of the two is compared on the test set with the behavior recognition network trained without the generated sample,

[0063] After comparison, if the accuracy is significantly improved, it shows that the generated new sample can improve the recognition accuracy of the behavior recognition network, and the technical requirements of the application are met, so as to realize the purpose of accurate recognition, that is, success.

[0064] Experimental verification:

[0065] 1) Human skeleton generation method test.

[0066] According to steps 1 and 2, the actions in the NTURGBD-60 data set are sorted, which contains 60 kinds of human behavior actions, of which the first 49 kinds are single-person behavior actions, and the last 11 kinds are double-person behavior actions, which basically cover most human behaviors in life.

[0067] Under the NTURGB-60 data set, according to table 1, the action data is divided to design the generation experiment, each experiment tests the generation experiment of retaining 10%, 20%, 30%, and 50% samples, wherein the class marked with * represents a small number of action categories that need to be expanded, and the class without * represents a complete action category that needs to be referred to.

[0068] Table 1, generation experiment grouping

[0069]

[0070] Some generation effects are shown in Figure 6a 、 Figure 6b , Figure 6a The drinking water generated by learning the style of combing hair for drinking water action, Figure 6b The double-foot jump generated by learning the double-foot jump action for single-foot jump.

[0071] From Figure 6a , Figure 6b It can be seen that the generated samples are difficult to distinguish from the original actions, but have slightly different styles, and thus can be used for data sample expansion.

[0072] 2) Behavior recognition method test.

[0073] In order to verify the effectiveness of the generated skeleton data for improving the behavior recognition network, the present application uses the generated new samples to test the performance of the currently disclosed behavior recognition network STGCN, PBGCN, and the specific results are shown in Table 2.

[0074] As can be seen from Table 2, for most cases, the behavior recognition network trained by adding the generated new samples has improved recognition rate on the test set; for the case where the deletion ratio is not greater than 50%, since the remaining samples have represented the style of most original samples, the experimental results are not ideal, which shows that there is still room for improvement of the network.

[0075] Table 2, experimental results of the generated new samples participating in the behavior recognition network

[0076]

Claims

1. A method for generating a human skeleton for behavior recognition, characterized in that, Follow these steps: Step 1: Collect relevant datasets of human skeletal movements; Step 2: Preprocess the dataset obtained in Step 1, select the action category samples to be generated and the complete action samples to be referenced in terms of style, and divide them into training set and test set; Step 3: Construct a generative network with skeletal style transfer features. The generator network consists of an encoder, a decoder, and a discriminator. The encoder and decoder together constitute the generator, and the encoder is further divided into a content encoder and a style encoder. The specific process is as follows: The velocity of the skeleton data in the NTURGBD60 dataset that will be used to train the generative network is calculated. The velocity is obtained by subtracting the coordinates of adjacent frames, as shown in Equation (1). The joint coordinates are input into the content encoder, and the velocity value is input into the style encoder. Then: （1） in, These are the coordinates of the joints in the human body. To calculate the velocity value at that moment, t The current moment; In the structure of the content encoder, the input data is human skeletal data. , T The first dimension representing the time dimension T The frame contains three values ​​representing the coordinates in the X, Y, and Z directions, respectively. The input coordinates are processed by a module consisting of three 1D convolutions with a stride of 2 and a BN layer to obtain the final bone vector. In addition, a BN normalization layer is used to speed up network convergence, making it less prone to divergence and improving training accuracy and speed. In the structure of the style encoder, the input data is skeletal data. , T The first dimension representing the time dimension T The frame, and the other three values ​​represent the velocity values ​​in the X, Y, and Z directions respectively. The velocity values ​​are calculated from the original data using equation (1). The input velocity values ​​are passed through a module with three 1D convolutions and BN layers connected in series with a stride of 1, and then through an LSTM layer to obtain the final velocity vector. The specific LSTM calculation values ​​are shown in equation (2). （2） in, Indicates the network parameters to be trained. Represents the feature value of the current node. This represents the accumulated feature values ​​from previous time steps. Represents the bias eigenvalue. Represented as a non-linear activation function; In the decoder structure, the content vectors and velocity vectors obtained from the two encoders are used as inputs. The content vectors are the main body and are processed through three concatenated 1D convolutions, SPAdaIN residual blocks, LSTM network layers, and one 1D convolution to obtain the skeletal data Y. The velocity vectors are processed through three fully connected Linear layers and then SPAdaIN operations are performed with the features of each stage of the content vectors to obtain the features of the final generated samples after correction. The specific calculation of the SPAdaIN operation is shown in Equation (3). （3） in, x, v These represent the input content vector and style vector, respectively. , These represent the mean and standard deviation of the corresponding features, respectively. The discriminator is composed of multiple traditional 1D convolutions connected in series; From this point on, during the training process, the final generated skeletal data is fed together with the skeletal data used in training the generative network and fed into the discriminator to obtain the complete generative network. Step 4: Train the content encoder, style encoder, decoder, and discriminator sequentially using the training set; Step 5: Use the generated new samples to train the behavior recognition network, improve the generalization ability of the behavior recognition network, and improve the accuracy of actions on the test set.

2. The method for generating human skeletons for behavior recognition according to claim 1, characterized in that, In step 1, the specific process is as follows: Choose from the UCF101, NTURGBD60, or NTURGBD60 datasets for the task of generating the skeleton; alternatively, use a custom dataset created with Kinetics sensors to obtain the joint coordinates of the wearable device. The collected data should be categorized and labeled with the types of actions for training purposes.

3. The method for generating human skeletons for behavior recognition according to claim 1, characterized in that, Step 2, the specific process is as follows: First, select the action category data to be generated and the complete distribution action data for reference; Secondly, samples with large errors are manually removed, and the data is normalized. For each group of samples, the coordinates of all joint points are divided by the maximum value so that the coordinates of the joint points are within the range of [0,1]. Then, the relative coordinates of the skeleton are converted into absolute coordinate values. That is, a certain joint point of the human body is used as the origin of the coordinates, and a translation operation is performed. The coordinates of the reference joint point are subtracted from the coordinates of each dimension of all joint points. Finally, the training set and the test set are divided.

4. The method for generating human skeletons for behavior recognition according to claim 1, characterized in that, Step 4, the specific process is as follows: During training, L1 distance reconstruction loss and W distance are used as loss functions. 4.1) Initialize training parameters, The training parameters are set as follows: epoch is the number of times the training traverses all data, set to an integer between 500 and 800; batch_size is the number of samples in each batch, set to one of {8, 16, 32, 64}; learning_rate is the learning rate, with an initial learning rate of 0.0001. 4.2) Train the generative network using the encoding / decoding reconstruction loss. The velocity data is calculated from the skeletal data of a small number of input movements according to Equation (1). The skeletal data of a small number of movements is input into the content encoder to obtain the content vector. The corresponding velocity data is input into the style encoder to obtain the style vector. The two content vectors and the style vector are then input into the decoder. At this time, the input is the original content and style. Therefore, the output of the decoder corresponds to the original skeletal data. The loss function adopts the reconstruction loss of L1 distance, as shown in Equation (4). ；（4） 4.3) Train the generative network using adversarial loss. Repeat step 4.2) with the complete skeletal data that needs to be referenced to obtain the reference content vector and reference style vector. Extract the required reference style vector and input it together with the original content vector into the decoder to obtain the new skeletal data. At this time, the generative network has the ability to generate samples. Since the generated samples have no standard for comparison, adversarial loss is used. The generated samples and the original samples are input together into the discriminator to distinguish between real and fake scores. Adversarial training of the generative network is performed. The W distance loss function is shown in equations (5) and (6): （5） （6） in, This indicates the generated new sample. This represents the score of the input data after passing through the discriminator. The coefficient corresponding to the W distance is dynamically adjusted as a hyperparameter.

5. The method for generating a human skeleton for behavior recognition according to claim 1, characterized in that, Step 5, the specific process is as follows: The behavior recognition network selects either the STGCN network or the PBGCN network to generate new samples. The accuracy of the behavior recognition network trained with the generated samples is compared with that trained without the generated samples on the test set. If the accuracy is significantly improved, it means that the generated samples can improve the recognition accuracy of the behavior recognition network and achieve accurate recognition.

Images