Human body posture video evaluation method based on space-time diagram convolutional network

Abstract

The invention relates to a human body posture video evaluation method based on a space-time diagram convolutional network, and belongs to the technical field of computer vision, and the method comprises the following steps: obtaining a human body posture point cloud video, carrying out the preprocessing of the video, and constructing a data set; then, an STGCN model is improved, so that the STGCN model can generate a human body posture evaluation score with high accuracy; the improved STGCN model is trained, and a finally trained network state is stored; and evaluating the human body posture video by using the trained generative network. According to the method, the high-accuracy human body posture evaluation score can be generated, the actual application scene is better met, and an important guiding effect on autonomous rehabilitation evaluation can be achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method for evaluating human posture videos based on a spatiotemporal graph convolutional network. Background Art

[0003] There are many types of existing autonomous rehabilitation assessment methods, including traditional methods such as machine learning and convolutional neural networks, but most of them cannot achieve high-accuracy assessment scores. They are usually limited to behavioral classification or rating, which makes it impossible to provide clear guidance on the user's body movements to help the user perform the assessment, thus affecting the actual application effect of the assessment.

[0004] Therefore, a new technical solution is urgently needed in the prior art to solve this problem. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a human posture video evaluation method based on a spatiotemporal graph convolutional network to solve the technical problem that existing autonomous rehabilitation evaluation methods are usually limited to behavior classification or grade assessment, cannot achieve highly accurate evaluation scores, and cannot provide clear guidance on body movements.

[0006] A human posture video assessment method based on a spatiotemporal graph convolutional network comprises the following steps, and the following steps are performed in sequence;

[0007] Step S1: Obtain a human body posture point cloud video, pre-process the key point data of human body posture recognition in the cloud video, and score the degree of movement of the key points of the subject in the cloud video, store the pre-processed key point human body posture data and the corresponding score value, and construct a human body posture dataset;

[0008] Step S2: Use the improved graph adjacency matrix construction method to construct the graph structure relationship between the key points in the human posture dataset and merge it with the preprocessed human posture data to form a new dataset. The new dataset is used as the input signal of the improved spatiotemporal graph convolutional network STGCN for motion feature extraction.

[0009] The improved spatiotemporal graph convolutional network STGCN is referred to as the improved STGCN model, including:

[0010] Add the DropBlockSke layer to the temporal feature extraction network part of the spatiotemporal graph convolutional network STGCN, and put the optimized feature vector result into the action evaluation network part;

[0011] Add a skeleton prediction network to the spatiotemporal feature extraction network of the spatiotemporal graph convolutional network STGCN to perform time series prediction of key points;

[0012] Step S3: Use the human body posture data set to train the improved STGCN model until the improved STGCN model reaches a low error, i.e., high accuracy indicator;

[0013] Step S4: Use the trained network to evaluate the human posture video and obtain the predicted value of the corresponding evaluation score.

[0014] The preprocessing in step S1 is: dividing the data into multiple batches, and each batch contains a four-dimensional feature vector N*T*V*C of 100 time steps, where N represents the size of each batch, T represents the time series length of each sample, V represents the number of key points, and C represents the number of feature channels.

[0015] In step S2, an improved method for constructing a graph adjacency matrix is ​​used to construct a graph structure relationship between joint points in the human posture dataset. The specific steps are as follows:

[0016] Step S2-1-1, inputting a human posture data set, mapping the key point data in the data set into nodes in a graph structure, and dividing these nodes;

[0017] Step S2-1-2, perform node partitioning improvement; divide a neighborhood of a node into two subsets, the first subset is the node itself and the set of neighboring nodes that are closer to the center of gravity of the entire skeleton than the node in spatial position, and the second subset is the set of neighboring nodes that are farther away from the center of gravity of the skeleton, thereby generating two adjacency matrices;

[0018] Step S2-1-3, normalize the two generated adjacency matrices by the inverse of the square root of the degree, where the normalization formula is as follows:

[0019]

[0020] Where A is the adjacency matrix of the graph; D is the degree matrix of the graph, which is a diagonal matrix; is the square root of the inverse of the degree matrix D.

[0021] The specific method of adding the DropBlockSke layer in step S2 to obtain the optimized feature vector result is:

[0022] Apply a binary mask to the input tensor to randomly discard features in certain areas to help the model prevent overfitting. The formula for DropBlockSke is as follows:

[0023]

[0024] M seed=cast(random.u(tf.shape(input_abs)) <input_abs_nor×gamma) (3-1);

[0025] M=matmul(M seed ,A) (3-2);

[0026] mask=expand_dims(expand_dims(1-M,axis=0),axis=0) (4);

[0027]

[0028] In formula (2), α is a constant related to the key point num_point. The number of key points num_point in the present invention is 25, so α in the present invention is 1.92; keep_prob represents the probability of retaining the node;

[0029] In formula (3-1), input_abs is the input feature vector of the layer, which represents the features or attributes of the nodes in the graph; input_abs_norm is the normalized result of input_abs, which is used to ensure that the values ​​of different features or nodes are in the same range; gamma is a parameter that controls sparsity. The parameter types calculated by formula (2) are all float32, and finally a two-dimensional mask matrix M is generated. seed ;

[0030] In formula (3-2), M_seed is the sparse mask matrix generated by formula (3-1), matmul is the matrix multiplication operation; M represents M seed The result of multiplication with the A matrix is ​​to expand M_seed to the graph structure;

[0031] In formula (4), 1-M means changing 0 in M ​​to 1 and 1 to 0 to generate a mask matrix; expand_dims(axis=0) is to add a dimension to the specified dimension; mask is the mask matrix finally generated to mask the input data;

[0032] In formula (5), iputs represents the input data; size(mask) represents the total number of elements in the mask; reduce_sum(mask) represents the total number of non-zero elements in the mask; output is the final output of the DropBlockSke layer, which is achieved by multiplying the input data with the mask to achieve block dropping.

[0033] The specific steps of performing time series prediction of key points in step S2 are:

[0034] The output of the added skeleton prediction network is used as the input of the self-supervised loss function to form a new loss function This loss function It is composed of Huber function and MSE function, and this loss function is minimized to optimize the spatiotemporal feature extraction network;

[0035] Among them, the skeleton prediction network includes LSTM layer, Dropout layer and TimeDistributed layer;

[0036] The loss function The formula is as follows:

[0037]

[0038] in,

[0039] In formula (6), Encoded i Indicates the predicted key point coordinate value of the current time step, seq_input_lt i Indicates the real key point coordinate value; L (mse) represents the mean square error, which is used as the self-supervised loss function here;

[0040] In formula (7), L huber Represents the Huber loss function, which is the loss function used in traditional model evaluation; represents the predicted value of the evaluation score; h i represents the true value of the evaluation score, which comes from the score value of the subject in the human posture dataset constructed in step S1; δ is a threshold used to control the behavior of the loss function.

[0041] In the process of training the improved STGCN model in step S3, the training process of the spatiotemporal feature extraction network part and the action assessment head network part are both trained using the loss function To express:

[0042]

[0043] Through the above design scheme, the present invention can bring the following beneficial effects:

[0044] 1. The graph node partitioning method provided by the present invention is aimed at the generation of the graph adjacency matrix, strengthens the connection relationship of the graph node structure relationship brought about by the node partitioning method of the current model, and improves the accuracy of model evaluation.

[0045] 2. The DropBlockSke layer provided by the present invention can help the model prevent overfitting when the model performs temporal convolution feature extraction, thereby improving the accuracy of model evaluation.

[0046] 3. Joint loss function provided by the present invention The output of the added skeleton prediction network is used as the self-supervised loss function L (mse) The input and L (huber) The combination of these functions helps the model achieve self-supervision and improves the accuracy of model evaluation.

[0047] In summary, the present invention uses posture scoring to evaluate human posture, which improves the accuracy of existing human posture video evaluation and can be promoted in fields such as human posture evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flow chart of a human posture video assessment method based on a spatiotemporal graph convolutional network according to the present invention;

[0049] Figure 2 It is a schematic diagram of the structure of the improved spatiotemporal graph convolutional network STGCN in the present invention;

[0050] Figure 3 This is a flow chart of the node division method adopted by the present invention;

[0051] Figure 4 It is a schematic diagram of the network structure for extracting spatiotemporal features of the present invention;

[0052] Figure 5 Schematic diagram of the network structure for extracting time features of the present invention

[0053] Figure 6 This is a flow chart of the DropBlockSke module used in the present invention.

[0054] Figure 7 This is a diagram of the skeleton prediction network structure used in the present invention.

[0055] Figure 8 This is the training and verification loss diagram of the improved STGCN model of the present invention.

[0056] Fig. 9 It is the index value of the improved STGCN model of the present invention. DETAILED DESCRIPTION

[0057] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a human posture video assessment method based on a spatiotemporal graph convolutional network of the present invention in conjunction with the accompanying drawings.

[0058] The present invention first uses an improved node partitioning method to strengthen the relationship between graph nodes, then adds the DropBlockSke module proposed by the present invention to the spatiotemporal feature extraction network to prevent model overfitting, and finally adds the skeleton prediction network proposed by the present invention to the network to construct a new loss function. The self-supervised learning mechanism of the model is realized, which effectively improves the accuracy of human posture assessment.

[0059] Figure 1 A schematic flow chart of a method for evaluating a human posture video based on a spatiotemporal graph convolutional network is provided. The specific steps of the method are as follows:

[0060] S1: Obtain human body posture point cloud video, preprocess it, and construct a data set;

[0061] S1-1, obtaining a human body posture point cloud video, wherein the human body posture point cloud video may be a depth video or an RGB video obtained by a depth sensor, or may be a public data set. The present invention adopts the public data set KIMORE.

[0062] S1-2, preprocess the human body posture point cloud video dataset, divide it into multiple batches, and each batch contains the four-dimensional feature vector N*T*V*C of 100 time steps to construct the human body posture dataset.

[0063] S2: Use the constructed data set and the improved method of constructing the graph adjacency matrix to construct a new graph adjacency matrix, and merge the human posture data set constructed in step S1-2 with it to form a new data set, and use the new data set as the input signal of the improved spatiotemporal graph convolutional network STGCN to extract action features; wherein, the structural schematic diagram of the improved spatiotemporal graph convolutional network STGCN is as follows Figure 2 shown.

[0064] S2-1, the data in the new dataset is a four-dimensional vector N*T*V*C, which is put into the spatiotemporal feature extraction network part of the improved spatiotemporal graph convolutional network STGCN for spatial and temporal feature extraction.

[0065] In step S2-1, the spatial configuration division in the graph adjacency matrix method is improved by neighbor node division, and the specific steps are as follows:

[0066] S2-1-1, input the human posture data set constructed in step S1-2 and perform node division.

[0067] S2-1-2, the improved node partitioning strategy is to divide the neighborhood of a node into two subsets, the first subset is the node itself and the set of neighboring nodes that are closer to the center of gravity of the entire skeleton than the node in spatial position, and the second subset is the set of neighboring nodes that are farther away from the center of gravity. In this way, two adjacency matrices are generated.

[0068] S2-1-3, normalize the two generated adjacency matrices by the inverse of the square root of the degree. The normalization formula is as follows:

[0069]

[0070] In formula (1), A is the adjacency matrix of the graph; D is the degree matrix of the graph, which is a diagonal matrix; is the square root of the inverse of the degree matrix D.

[0071] The normalized adjacency matrix obtained in this way is A Making the sum of the adjacency weights of each node equal to 1 helps better capture the relationship between nodes in graph convolutional networks. This process is often called symmetric normalization.

[0072] Figure 3 A flow chart of a node partitioning method proposed according to an example of the present invention is presented.

[0073] S2-2, add the DropBlock module to the temporal feature extraction network part, and after obtaining the optimized feature vector result, put the feature vector into the action evaluation head network part.

[0074] Figure 4 The spatiotemporal feature extraction network proposed in the example of the present invention is presented. Figure 5 The temporal feature extraction network proposed in the present invention is presented.

[0075] In step S2-2, the temporal feature extraction network is improved. A variant of the DropBlock technique is used in the temporal convolution part of the temporal feature extraction network, namely a DropBlockSke for key point estimation task, which mainly targets image data, usually four-dimensional data, and randomly discards features of certain areas by applying a binary mask to the input tensor to help the model prevent overfitting, and merges this result with the feature vector to obtain the final feature extraction result vector.

[0076] The specific calculation flow chart of DropBlockSke of the present invention is as follows: Figure 6 As shown, the formula for calculating DropBlockSke is as follows:

[0077]

[0078] M seed =cast(random.u(shape(input_abs)) <input_abs_norm×gamma) (3-1);

[0079] M=matmul(M seed ,A) (3-2);

[0080] mask=expand_dims(expand_dims(1-M,axis=0),axis=0) (4);

[0081]

[0082] In formula (2), α is a constant related to num_point. The number of key points num_point in the present invention is 25, so α in the present invention is 1.92, and keep_prob represents the probability of retaining the node;

[0083] In formula (3-1), input_abs is the input feature vector of the layer, which represents the features or attributes of the nodes in the graph. input_abs_norm is the normalized result of input_abs, which is used to ensure that the values ​​of different features or nodes are in the same range. gamma is a parameter that controls sparsity, and the threshold is calculated by formula (2). The data types of these parameters are all float32, and finally a two-dimensional mask matrix M is generated. seed ;

[0084] In formula (3-2), M_seed is the sparse mask matrix generated by formula (3-1), A represents the adjacency matrix of the graph, matmul is the matrix multiplication operation, and M represents M seed The result obtained by multiplying the matrix A. This step can be regarded as extending M_seed to the graph structure;

[0085] In formula (4), 1-M means changing 0 in M ​​to 1 and 1 to 0 to generate a mask matrix, expand_dims(axis=0) means adding a dimension to the specified dimension, and mask is the final mask matrix used to mask the input data;

[0086] In formula (5), mask is the mask matrix generated by formula (4), inputs represents the input data, size(mask) represents the total number of elements in the mask, reduce_sum(mask) represents the total number of non-zero elements in the mask, and output is the final output of the DropBlockSke layer. Output is achieved by multiplying the input data with mask to achieve block dropping.

[0087] S2-3, improve the spatiotemporal feature extraction network part, add the skeleton prediction network, and obtain the optimized network for training.

[0088] In step S2-3, the output of the added skeleton prediction network is used as the input to the self-supervised loss function. In this paper, the mean square error L (mse) As a self-supervised loss function, the prediction results generated by this module are used to guide the training process of the network. Based on this strategy, a new loss function is designed This helps the model realize the self-supervised learning mechanism. This loss function combines the Huber function and the MSE function to jointly guide the training process of this network, aiming to further optimize the performance of the action evaluation network. Specifically, the loss function proposed in this invention is The fusion strategy of Huber function and MSE function can make full use of the characteristics of Huber function, making the model more robust in the face of outliers and avoiding excessive fluctuations in the loss function caused by outliers. At the same time, the introduction of MSE function can also more accurately guide the model to gradually reduce the error during the training process.

[0089] The loss function proposed by the present invention The calculation is as follows:

[0090]

[0091]

[0092]

[0093] In formula (6), Encoded i Indicates the predicted key point coordinate value of the current time step, seq_input_lt i Indicates the real key point coordinate value.

[0094] In formula (7), L is defined huber , L huber Represents the Huber loss function, which is the loss function used in the original model evaluation; represents the predicted value of the evaluation score, which is the output value of the final step S4 of the present invention; i represents the true value of the evaluation score, which comes from the score value of the subject in the human posture dataset constructed in step S1; δ is a threshold used to control the behavior of the loss function.

[0095] In formula (8), is the loss function proposed by the present invention, L huberFrom formula (7), we can get L mse It is obtained by formula (6).

[0096] The skeleton prediction network of the present invention is specifically as follows Figure 7 shown.

[0097] The skeleton prediction network of the present invention is composed of LSTM stacked twice. It includes two LSTM layers, Dropout layer and TimeDistributed layer. First, the output Y of STGCN block is l As input, the data is reshaped into a two-dimensional shape, i.e. (-1, x.shape[2]*x.shape[3]), through the Reshape layer, which effectively integrates the time step and key point information. Next, the data passes through a 450-unit LSTM layer and a Dropout layer is applied to prevent overfitting. Then, the TimeDistributed layer is connected to replicate the information in the time dimension, so that the model can better handle sequence data. Next, the data is reshaped again and enters the second LSTM layer with 1000 units to further extract the timing information. Similarly, after applying the Dropout and TimeDistributed layers again, the data is connected to a fully connected layer with 75 neurons to map to the output space. Finally, the coordinate predictions of 25 key points (3-channel information for each key point) are generated. This structure is designed to process key point time series data, learn the timing information in the network and generate predictions for the key point positions.

[0098] S3: Train the improved STGCN network model until the improved STGCN model reaches a low error or high accuracy indicator;

[0099] In step S3, the improved STGCN model is trained, the number of iterations is set to 2000, the Adam learning rate defaults to 0.0001, the batch size batch_size is set to 10, and the self-supervised loss function is used. During the training process, the training process of the feature extraction network part and the action evaluation network part can be expressed as:

[0100]

[0101] The meaning of formula (8) has been explained above and will not be described here.

[0102] The training loss and verification loss results of the present invention are as follows Figure 8 As shown, we can see that the loss function and convergence speed of this model are very good.

[0103] The indicators used in the present invention are common indicators for measuring the performance of prediction models:

[0104] Mean Absolute Deviation (MAD): is an indicator to measure the prediction error, which represents the average value of the absolute error between the observed value and the predicted value. For each sample, the absolute difference between its predicted value and the actual value is calculated, and then the average value of all samples is taken. That is, the smaller the MAD, the more accurate the model's prediction. The calculation formula is as follows:

[0105]

[0106] In formula (9), Y i represents the true value, represents the predicted value, and n represents the number of samples.

[0107] Root Mean Square Error (RMSE): It is the root mean square value of the difference between the observed value and the predicted value, that is, the square root of the average value of the square error. Similar to MAD, the smaller the RMSDeviation, the more accurate the model's prediction. The calculation formula is as follows:

[0108]

[0109] Mean Absolute Percentage Error (MAPE): is a percentage indicator that represents the average value of the percentage error between the observed value and the predicted value. That is, the smaller the MAPE, the lower the percentage error of the model. The calculation formula is as follows:

[0110]

[0111] The three indicators of this model are as follows Fig. 9 shown.

[0112] Table 1 is a comparison table of this model and the existing posture assessment model

[0113]

[0114] The numerical values ​​and model details of these comparison models are published in:

[0115] [1]Deb, Swakshar, et al. "Graph convolutional networks for assessment of physical rehabilitation exercises." IEEE Transactions on NeuralSystemsandRehabilitation Engineering30(2022):410-419.

[0116] [2]YFSong, Z.Zhang, C.Shan, and L.Wang, "Richlyactivated graphconvolutional network for robust skeleton-based action recognition," IEEETrans.Circuits Syst.Video Technol., vol.31, no.5, pp.1915–1925, May2021.

[0117] [3] P. Zhang, C. Lan, W. Zeng, J.

[0118] [4] Y.Liao, A.Vakanski, and M.Xian, "A deep learning framework for assessing physical rehabilitation exercises," IEEE Trans.NeuralSyst.Rehabil.Eng., vol.28, no.2, pp.468–477, Feb.2020.

[0119] [5] S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in Proc.AAAIConf.Artif.Intell., Apr.2018, vol.32, no.1, pp.1–10.

[0120] S4: Use the trained network to evaluate human pose videos.

[0121] In summary, the accuracy of this model is higher than that of the existing posture assessment model.

[0122] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A human posture video assessment method based on spatiotemporal graph convolutional network, characterized by: The method comprises the following steps, and the following steps are performed in sequence: Step S1: Obtain a human body posture point cloud video, pre-process the key point data of human body posture recognition in the cloud video, and score the degree of movement of the key points of the subject in the cloud video, store the pre-processed key point human body posture data and the corresponding score value, and construct a human body posture dataset; Step S2: Use the improved graph adjacency matrix construction method to construct the graph structure relationship between the key points in the human posture dataset and merge it with the preprocessed human posture data to form a new dataset. The new dataset is used as the input signal of the improved spatiotemporal graph convolutional network STGCN for motion feature extraction. The improved spatiotemporal graph convolutional network STGCN is referred to as the improved STGCN model, including: Add the DropBlockSke layer to the temporal feature extraction network part of the spatiotemporal graph convolutional network STGCN, and put the optimized feature vector result into the action evaluation network part; Add a skeleton prediction network to the spatiotemporal feature extraction network of the spatiotemporal graph convolutional network STGCN to perform time series prediction of key points; Step S3: Use the human body posture data set to train the improved STGCN model until the improved STGCN model reaches a low error, i.e., high accuracy indicator; Step S4: Use the trained network to evaluate the human posture video and obtain the predicted value of the corresponding evaluation score.

2. According to claim 1, a human posture video assessment method based on spatiotemporal graph convolutional network is characterized by: The preprocessing in step S1 is: dividing the data into multiple batches, and each batch contains a four-dimensional feature vector N*T*V*C of 100 time steps, where N represents the size of each batch, T represents the time series length of each sample, V represents the number of key points, and C represents the number of feature channels.

3. The method for human posture video assessment based on spatiotemporal graph convolutional network according to claim 1, characterized in that: In step S2, an improved method for constructing a graph adjacency matrix is ​​used to construct a graph structure relationship between joint points in the human posture dataset. The specific steps are as follows: Step S2-1-1, inputting a human posture data set, mapping the key point data in the data set into nodes in a graph structure, and dividing these nodes; Step S2-1-2, perform node partitioning improvement; divide a neighborhood of a node into two subsets, the first subset is the node itself and the set of neighboring nodes that are closer to the center of gravity of the entire skeleton than the node in spatial position, and the second subset is the set of neighboring nodes that are farther away from the center of gravity of the skeleton, thereby generating two adjacency matrices; Step S2-1-3, normalize the two generated adjacency matrices by the inverse of the square root of the degree, where the normalization formula is as follows: Where A is the adjacency matrix of the graph; D is the degree matrix of the graph, which is a diagonal matrix; is the square root of the inverse of the degree matrix D.

4. The method for human posture video assessment based on spatiotemporal graph convolutional network according to claim 1, characterized in that: The specific method of adding the DropBlockSke layer in step S2 to obtain the optimized feature vector result is: Apply a binary mask to the input tensor to randomly discard features in certain areas to help the model prevent overfitting. The formula for DropBlockSke is as follows: M seed =cast(random.u(tf.shape(input_abs))<input_abs_nor×gomma) (3-1); M=matmul(M seed ,A) (3-2); mask=expand_dims(expand_dims(1-M,axis=0),axis=0) (4); In formula (2), α is a constant related to the key point num_point. The number of key points num_point in the present invention is 25, so α in the present invention is 1.92; keep_prob represents the probability of retaining the node; In formula (3-1), input_abs is the input feature vector of the layer, which represents the features or attributes of the nodes in the graph; input_abs_norm is the normalized result of input_abs, which is used to ensure that the values ​​of different features or nodes are in the same range; gamma is a parameter that controls sparsity. The parameter types calculated by formula (2) are all float32, and finally a two-dimensional mask matrix M is generated. seed ; In formula (3-2), M_seed is the sparse mask matrix generated by formula (3-1), matmul is the matrix multiplication operation; M represents M seed The result of multiplying the matrix A is to expand M_seed to the graph structure; In formula (4), 1-M means changing 0 in M ​​to 1 and 1 to 0 to generate a mask matrix; expand_dims(axis=0) is to add a dimension to the specified dimension; mask is the mask matrix finally generated to mask the input data; In formula (5), inputs represents the input data; size(mask) represents the total number of elements in the mask; reduce_sum(mask) represents the total number of non-zero elements in the mask; output is the final output of the DropBlockSke layer, which is achieved by multiplying the input data with msak to achieve block dropping.

5. The method for human posture video assessment based on spatiotemporal graph convolutional network according to claim 1, characterized in that: The specific steps of performing time series prediction of key points in step S2 are: The output of the added skeleton prediction network is used as the input of the self-supervised loss function to form a new loss function This loss function It is composed of Huber function and MSE function, and this loss function is minimized to optimize the spatiotemporal feature extraction network; Among them, the skeleton prediction network includes LSTM layer, Dropout layer and TimeDistributed layer; The loss function The formula is as follows: in, In formula (6), Encoded i Indicates the predicted key point coordinate value of the current time step, seq_input_lt i Indicates the real key point coordinate value; L (mse) represents the mean square error, which is used as the self-supervised loss function here; In formula (7), L huber Represents the Huber loss function, which is the loss function used in traditional model evaluation; represents the predicted value of the evaluation score; h i represents the true value of the evaluation score, which comes from the score value of the subject in the human posture dataset constructed in step S1; δ is a threshold used to control the behavior of the loss function.

6. The method for human posture video assessment based on spatiotemporal graph convolutional network according to claim 1, characterized in that: In the process of training the improved STGCN model in step S3, the training process of the spatiotemporal feature extraction network part and the action assessment head network part are both trained using the loss function To express:

Images