A Human Posture Video Assessment Method Based on Spatiotemporal Graph Convolutional Network
Through the human posture video evaluation method based on spatiotemporal graph convolutional network, the problem of inaccurate evaluation scores in the existing technology is solved. Through the improved STGCN model and self-supervised loss function, high-accuracy limb movement evaluation is achieved, and the model's connectivity and overfitting prevention capabilities are enhanced.
Patent Information
- Application Number
- CN202411650835.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2044-11-19
AI Technical Summary
Existing autonomous rehabilitation assessment methods are usually limited to behavioral classification or rating, which cannot achieve highly accurate assessment scores and cannot provide clear guidance on the user's physical movements.
A human posture video evaluation method based on spatiotemporal graph convolutional network is adopted. By obtaining human posture point cloud video, preprocessing and key point data scoring are performed, and graph structure relationships are constructed. The improved STGCN model is used for motion feature extraction, and the DropBlockSke layer and skeleton prediction network are added. The training is combined with the self-supervised loss function to optimize the model evaluation.
It improves the accuracy of human posture video assessment, can more accurately guide the user's body movement assessment, and enhances the model's connectivity and ability to prevent overfitting.
Smart Images
Figure CN119693999B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method for evaluating human posture videos based on a spatiotemporal graph convolutional network. Background Art
[0002] Existing autonomous rehabilitation assessment methods vary widely, including traditional approaches like machine learning and convolutional neural networks. However, most fail to achieve highly accurate assessment scores. These methods are often limited to behavioral classification or rating, making it difficult to provide clear guidance on user movements to aid assessment, thus hindering the effectiveness of the assessment.
[0003] Therefore, a new technical solution is urgently needed in the existing technology to solve this problem. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to provide a human posture video evaluation method based on a spatiotemporal graph convolutional network to solve the technical problem that existing autonomous rehabilitation evaluation methods are usually limited to behavior classification or grade assessment, cannot achieve highly accurate evaluation scores, and cannot provide clear guidance on limb movements.
[0005] A method for human posture video assessment based on spatiotemporal graph convolutional network comprises the following steps, which are performed in sequence:
[0006] Step S1: Obtain a human body posture point cloud video, pre-process the key point data for human body posture recognition in the cloud video, score the degree of key point movement of the subject in the cloud video, store the pre-processed key point human body posture data and the corresponding score values, and construct a human body posture dataset;
[0007] Step S2: Use the improved graph adjacency matrix construction method to construct the graph structure relationship between the key points in the human posture dataset and merge it with the preprocessed human posture data to form a new dataset. The new dataset is used as the input signal of the improved spatiotemporal graph convolutional network STGCN for motion feature extraction;
[0008] The improved spatiotemporal graph convolutional network STGCN is referred to as the improved STGCN model, including:
[0009] Add the DropBlockSke layer to the temporal feature extraction network part of the spatiotemporal graph convolutional network STGCN, and then put the optimized feature vector result into the action evaluation network part;
[0010] A skeleton prediction network is added to the spatiotemporal feature extraction network of the spatiotemporal graph convolutional network STGCN to perform time series prediction of key points;
[0011] Step S3: Use the human posture dataset to train the improved STGCN model until the improved STGCN model reaches a low error or high accuracy index;
[0012] Step S4: Use the trained network to evaluate the human posture video and obtain the predicted value of the corresponding evaluation score.
[0013] The preprocessing in step S1 is as follows: the data is divided into multiple batches, and each batch contains a four-dimensional feature vector N*T*V*C of 100 time steps, where N represents the size of each batch, T represents the time series length of each sample, V represents the number of key points, and C represents the number of feature channels.
[0014] In step S2, an improved method for constructing a graph adjacency matrix is used to construct a graph structure relationship between joint points in the human posture dataset. The specific steps are as follows:
[0015] Step S2-1-1: input a human posture dataset, map the key point data in the dataset into nodes in a graph structure, and divide these nodes;
[0016] Step S2-1-2, perform node partitioning improvement; divide a neighborhood of a node into two subsets, the first subset is the node itself and the set of neighboring nodes that are spatially closer to the center of gravity of the entire skeleton than the node itself, and the second subset is the set of neighboring nodes that are farther away from the center of gravity of the skeleton, thereby generating two adjacency matrices;
[0017] Step S2-1-3: normalize the two generated adjacency matrices using the inverse of the square root of the degree, where the normalization formula is as follows:
[0018]
[0019] Where A is the adjacency matrix of the graph; D is the degree matrix of the graph, which is a diagonal matrix; is the square root of the inverse of the degree matrix D.
[0020] The specific method of adding the DropBlockSke layer in step S2 to obtain the optimized feature vector result is:
[0021] Apply a binary mask to the input tensor to randomly drop features in certain areas to help the model prevent overfitting. The formula for DropBlockSke is as follows:
[0022]
[0023] M seed=cast(random.u(tf.shape(input_abs)) <input_abs_nor×gamma) (3-1);
[0024] M=matmul(M seed ,A) (3-2);
[0025] mask=expand_dims(expand_dims(1-M,axis=0),axis=0) (4);
[0026]
[0027] In formula (2), α is a constant related to the key point num_point. The number of key points num_point in the present invention is 25, so α in the present invention is 1.92; keep_prob represents the probability of retaining the node;
[0028] In formula (3-1), input_abs is the input feature vector of the layer, which represents the features or attributes of the nodes in the graph; input_abs_norm is the normalized result of input_abs, which is used to ensure that the values of different features or nodes are in the same range; gamma is a parameter that controls sparsity. The parameter types calculated by formula (2) are all float32, and finally a two-dimensional mask matrix M is generated. seed ;
[0029] In formula (3-2), M_seed is the sparse mask matrix generated by formula (3-1), matmul is the matrix multiplication operation; M represents M seed The result of multiplying the matrix A is to expand M_seed to the graph structure;
[0030] In formula (4), 1-M means changing 0 in M to 1 and 1 to 0 to generate a mask matrix; expand_dims(axis=0) means adding a dimension to the specified dimension; mask is the final mask matrix used to mask the input data;
[0031] In formula (5), iputs represents the input data; size(mask) represents the total number of elements in the mask; reduce_sum(mask) represents the total number of non-zero elements in the mask; output is the final output of the DropBlockSke layer, which is achieved by multiplying the input data with the mask to achieve block dropping.
[0032] The specific steps of performing time series prediction of key points in step S2 are:
[0033] The output of the added skeleton prediction network is used as the input of the self-supervised loss function to form a new loss function This loss function It is composed of Huber function and MSE function, minimizing this loss function to optimize the spatiotemporal feature extraction network;
[0034] Among them, the skeleton prediction network includes LSTM layer, Dropout layer and TimeDistributed layer;
[0035] The loss function The formula is as follows:
[0036]
[0037] in,
[0038] In formula (6), Encoded i Indicates the predicted key point coordinate value of the current time step, seq_input_lt i Indicates the real key point coordinate value; L (mse) represents the mean square error, which is used here as the self-supervised loss function;
[0039] In formula (7), L huber Represents the Huber loss function, which is the loss function used in traditional model evaluation; represents the predicted value of the evaluation score; h i represents the true value of the evaluation score, which comes from the score value of the subjects in the human posture dataset constructed in step S1; δ is a threshold used to control the behavior of the loss function.
[0040] In the process of training the improved STGCN model in step S3, the training process of the spatiotemporal feature extraction network part and the action assessment head network part are both trained using the loss function To express:
[0041]
[0042] Through the above design scheme, the present invention can bring the following beneficial effects:
[0043] 1. The graph node partitioning method provided by the present invention strengthens the connection relationship of the graph node structure relationship brought about by the node partitioning method of the current model for the generation of the graph adjacency matrix, thereby improving the accuracy of model evaluation.
[0044] 2. The DropBlockSke layer provided by the present invention can help the model prevent overfitting when performing temporal convolution feature extraction, thereby improving the accuracy of model evaluation.
[0045] 3. Joint loss function provided by the present invention The output of the added skeleton prediction network is used as the self-supervised loss function L (mse) input, and with L (huber) The combination of these functions helps the model achieve self-supervision and improves the accuracy of model evaluation.
[0046] In summary, the present invention uses posture scoring to evaluate human posture, which improves the accuracy of existing human posture video assessment and can be promoted in fields such as human posture assessment. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 This is a flow chart of a method for human posture video assessment based on spatiotemporal graph convolutional network of the present invention;
[0048] Figure 2 Schematic diagram of the structure of the improved spatiotemporal graph convolutional network STGCN in the present invention;
[0049] Figure 3 This is a flow chart of the node division method adopted by the present invention;
[0050] Figure 4 This is a schematic diagram of the spatiotemporal feature extraction network structure of the present invention;
[0051] Figure 5 Schematic diagram of the time feature extraction network structure of the present invention
[0052] Figure 6 This is a flow chart of the DropBlockSke module used in the present invention.
[0053] Figure 7 This is a diagram of the skeleton prediction network structure used in the present invention.
[0054] Figure 8 This is the training and validation loss graph of the improved STGCN model of the present invention.
[0055] Figure 9 It is the index value of the improved STGCN model of the present invention. DETAILED DESCRIPTION
[0056] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a human posture video assessment method based on a spatiotemporal graph convolutional network of the present invention in conjunction with the accompanying drawings.
[0057] The present invention first uses an improved node partitioning method to strengthen the relationship between graph nodes, then adds the DropBlockSke module proposed in the present invention to the spatiotemporal feature extraction network to prevent model overfitting, and finally adds the skeleton prediction network proposed in the present invention to the network to construct a new loss function. The self-supervised learning mechanism of the model is implemented, which effectively improves the accuracy of human posture assessment.
[0058] Figure 1 The present invention provides a flow chart of a method for evaluating human posture videos based on a spatiotemporal graph convolutional network. The specific steps of the method are as follows:
[0059] S1: Obtain human body posture point cloud video, preprocess it, and construct a dataset;
[0060] S1-1, obtain a human body posture point cloud video, which can be a depth video or RGB video obtained by a depth sensor, or a public data set. The present invention uses the public data set KIMORE.
[0061] S1-2, preprocess the human body posture point cloud video dataset, divide it into multiple batches, and each batch contains the four-dimensional feature vector N*T*V*C of 100 time steps to construct the human body posture dataset.
[0062] S2: Use the constructed dataset and the improved graph adjacency matrix construction method to construct a new graph adjacency matrix, and merge the human posture dataset constructed in step S1-2 with it to form a new dataset, and use the new dataset as the input signal of the improved spatiotemporal graph convolutional network STGCN to extract motion features; Among them, the structural diagram of the improved spatiotemporal graph convolutional network STGCN is as follows Figure 2 shown.
[0063] S2-1, the data in the new dataset is a four-dimensional vector N*T*V*C, which is put into the spatiotemporal feature extraction network part of the improved spatiotemporal graph convolutional network STGCN for spatial and temporal feature extraction.
[0064] In step S2-1, the spatial configuration partitioning in the graph adjacency matrix method is improved by partitioning the neighbor nodes. The specific steps are as follows:
[0065] S2-1-1, input the human posture dataset constructed in step S1-2 and perform node division.
[0066] S2-1-2, the improved node partitioning strategy is to divide the neighborhood of a node into two subsets. The first subset consists of the node itself and the set of neighboring nodes that are spatially closer to the center of gravity of the entire skeleton than the node itself, and the second subset consists of the set of neighboring nodes that are farther away from the center of gravity. This generates two adjacency matrices.
[0067] S2-1-3, normalize the two generated adjacency matrices by the inverse of the square root of the degree. The normalization formula is as follows:
[0068]
[0069] In formula (1), A is the adjacency matrix of the graph; D is the degree matrix of the graph, which is a diagonal matrix; is the square root of the inverse of the degree matrix D.
[0070] The normalized adjacency matrix obtained in this way is normalized A Making the sum of the adjacency weights of each node equal to 1 helps better capture the relationship between nodes in graph convolutional networks. This process is often called symmetric normalization.
[0071] Figure 3 A flowchart of a node partitioning method proposed according to an example of the present invention is presented.
[0072] S2-2, add the DropBlock module to the temporal feature extraction network part, obtain the optimized feature vector result and put the feature vector into the action evaluation head network part.
[0073] Figure 4 The spatiotemporal feature extraction network proposed in the example of the present invention is presented. Figure 5 The temporal feature extraction network proposed in the present invention is presented.
[0074] In step S2-2, the temporal feature extraction network is improved. A variant of the DropBlock technique, DropBlockSke, is used in the temporal convolution portion of the temporal feature extraction network. This technique, specifically for key point estimation tasks, targets image data, typically four-dimensional data. It applies a binary mask to the input tensor to randomly discard features in certain regions to help the model prevent overfitting. This mask is then fused with the feature vector to produce the final feature extraction result vector.
[0075] The specific calculation flow chart of DropBlockSke of the present invention is as follows: Figure 6 As shown, the formula for calculating DropBlockSke is as follows:
[0076]
[0077] M seed =cast(random.u(shape(input_abs)) <input_abs_norm×gamma) (3-1);
[0078] M=matmul(M seed ,A) (3-2);
[0079] mask=expand_dims(expand_dims(1-M,axis=0),axis=0) (4);
[0080]
[0081] In formula (2), α is a constant related to num_point. The number of key points num_point in the present invention is 25, so α in the present invention is 1.92. keep_prob represents the probability of retaining the node.
[0082] In formula (3-1), input_abs is the input feature vector of the layer, which represents the features or attributes of the nodes in the graph. Input_abs_norm is the normalized result of input_abs, which is used to ensure that the values of different features or nodes are in the same range. Gamma is a parameter that controls sparsity. The threshold is calculated by formula (2). The data types of these parameters are all float32, and finally a two-dimensional mask matrix M is generated. seed ;
[0083] In formula (3-2), M_seed is the sparse mask matrix generated by formula (3-1), A represents the adjacency matrix of the graph, matmul is the matrix multiplication operation, and M represents M seed The result obtained by multiplying the matrix A can be regarded as extending M_seed to the graph structure;
[0084] In formula (4), 1-M means changing 0 in M to 1 and 1 to 0 to generate a mask matrix, expand_dims(axis=0) means adding a dimension to the specified dimension, and mask is the final mask matrix used to mask the input data;
[0085] In formula (5), mask is the mask matrix generated by formula (4), inputs represents the input data, size(mask) represents the total number of elements in the mask, reduce_sum(mask) represents the total number of non-zero elements in the mask, and output is the final output of the DropBlockSke layer. Output is the multiplication of the input data and mask to achieve block dropping.
[0086] S2-3, improve the spatiotemporal feature extraction network part, add the skeleton prediction network, and obtain the optimized network for training.
[0087] In step S2-3, the output of the added skeleton prediction network is used as the input to the self-supervised loss function. In this paper, the mean square error L (mse) It is trained as a self-supervised loss function, and the prediction results generated by this module are used to guide the training process of the network. Based on this strategy, a new loss function is designed. This helps the model realize the self-supervised learning mechanism. This loss function combines the Huber function and the MSE function to guide the training process of this network, aiming to further optimize the performance of the action evaluation network. Specifically, the loss function proposed in this invention is This fusion strategy, which combines the Huber function with the MSE function, fully leverages the characteristics of the Huber function, making the model more robust to outliers and avoiding excessive fluctuations in the loss function caused by outliers. Furthermore, the introduction of the MSE function more accurately guides the model to gradually reduce error during training.
[0088] The loss function proposed in this invention The calculation is as follows:
[0089]
[0090]
[0091]
[0092] In formula (6), Encoded i Indicates the predicted key point coordinate value of the current time step, seq_input_lt i Indicates the real key point coordinate value.
[0093] In formula (7), L is defined huber , L huber Represents the Huber loss function, which is the loss function used in the original model evaluation; The predicted value of the evaluation score is the output value of the final step S4 of the present invention; i represents the true value of the evaluation score, which comes from the score value of the subjects in the human posture dataset constructed in step S1; δ is a threshold used to control the behavior of the loss function.
[0094] In formula (8), is the loss function proposed by this invention, L huberAccording to formula (7), L mse Obtained by formula (6).
[0095] The skeleton prediction network of the present invention is specifically as follows Figure 7 shown.
[0096] The skeleton prediction network of the present invention is composed of LSTM stacked twice. It includes two LSTM layers, Dropout layer and TimeDistributed layer. First, the output Y of STGCN block is converted into l As input, the data is reshaped into a two-dimensional shape through the Reshape layer, i.e. (-1, x.shape[2]*x.shape[3]), effectively integrating the time step and key point information. Next, the data passes through a 450-unit LSTM layer, and the Dropout layer is applied to prevent overfitting. Then, the TimeDistributed layer is connected to replicate the information on the time dimension, so that the model can better process sequence data. Immediately after that, the data is reshaped again and enters the second LSTM layer with 1000 units to further extract the timing information. Similarly, after applying the Dropout and TimeDistributed layers again, the data is connected to a fully connected layer with 75 neurons to map to the output space. Finally, the coordinate predictions of 25 key points (3-channel information for each key point) are generated. This structure is designed to process key point time series data, learn the timing information in the network and generate predictions for the key point positions.
[0097] S3: Train the improved STGCN network model until the improved STGCN model reaches a low error or high accuracy indicator;
[0098] In step S3, the improved STGCN model is trained, the number of iterations is set to 2000, the Adam learning rate defaults to 0.0001, the batch size batch_size is set to 10, and the self-supervised loss function is used. During the training process, the feature extraction network part and the action evaluation network part can be expressed as follows:
[0099]
[0100] The meaning of formula (8) has been explained above and will not be described again here.
[0101] The training loss and verification loss results of the present invention are as follows Figure 8 As shown, we can see that the loss function of this model and its convergence speed are very good.
[0102] The indicators used in this invention are common indicators for measuring the performance of prediction models:
[0103] Mean Absolute Deviation (MAD): is a metric that measures prediction error. It represents the average of the absolute errors between the observed and predicted values. For each sample, the absolute difference between the predicted value and the actual value is calculated, and then the average of all samples is taken. The smaller the MAD, the more accurate the model's prediction. The calculation formula is as follows:
[0104]
[0105] In formula (9), Y i represents the true value, represents the predicted value, and n represents the number of samples.
[0106] Root Mean Square Error (RMSE): is the root mean square value of the difference between the observed value and the predicted value, that is, the square root of the average of the squared errors. Similar to MAD, the smaller the RMSDeviation, the more accurate the model's prediction. The calculation formula is as follows:
[0107]
[0108] Mean Absolute Percentage Error (MAPE): is a percentage indicator that represents the average of the percentage errors between the observed value and the predicted value. That is, the smaller the MAPE, the lower the percentage error of the model. The calculation formula is as follows:
[0109]
[0110] The three indicators of this model are as follows Figure 9 shown.
[0111] Table 1 is a comparison table of this model and existing posture assessment models
[0112]
[0113] The numerical values and model details of these comparison models are published in:
[0114] [1]Deb, Swakshar, et al. "Graph convolutional networks for assessment of physical rehabilitation exercises." IEEE Transactions on NeuralSystemsandRehabilitation Engineering30(2022):410-419.
[0115] [2]YFSong, Z.Zhang, C.Shan, and L.Wang, "Richlyactivated graphconvolutional network for robust skeleton-based action recognition," IEEETrans.Circuits Syst.Video Technol., vol.31, no.5, pp.1915–1925, May2021.
[0116] [3] P. Zhang, C. Lan, W. Zeng, J.
[0117] [4] Y.Liao, A.Vakanski, and M.Xian, "A deep learning framework for assessing physical rehabilitation exercises," IEEE Trans.NeuralSyst.Rehabil.Eng., vol.28, no.2, pp.468–477, Feb.2020.
[0118] [5] S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in Proc.AAAIConf.Artif.Intell., Apr.2018, vol.32, no.1, pp.1–10.
[0119] S4: Use the trained network to evaluate human pose videos.
[0120] In summary, the accuracy of this model is higher than that of the existing posture estimation model.
[0121] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.
Claims
1. A method for human posture video assessment based on spatiotemporal graph convolutional networks, characterized by: The following steps are included and are performed in sequence: Step S1: Obtain a human body posture point cloud video, pre-process the key point data for human body posture recognition in the cloud video, score the degree of key point movement of the subject in the cloud video, store the pre-processed key point human body posture data and the corresponding score values, and construct a human body posture dataset; Step S2: Use the improved graph adjacency matrix construction method to construct the graph structure relationship between the key points in the human posture dataset and merge it with the preprocessed human posture data to form a new dataset. The new dataset is used as the input signal of the improved spatiotemporal graph convolutional network STGCN for motion feature extraction; The improved spatiotemporal graph convolutional network STGCN is referred to as the improved STGCN model, including: Add the DropBlockSke layer to the temporal feature extraction network part of the spatiotemporal graph convolutional network STGCN, and then put the optimized feature vector result into the action evaluation network part; A skeleton prediction network is added to the spatiotemporal feature extraction network of the spatiotemporal graph convolutional network STGCN to perform time series prediction of key points; Among them, the improved method of constructing graph adjacency matrix is used to construct the graph structure relationship between the joint points in the human posture dataset. The specific steps are as follows: Step S2-1-1: input a human posture dataset, map the key point data in the dataset into nodes in a graph structure, and divide these nodes; Step S2-1-2, perform node partitioning improvement; divide a neighborhood of a node into two subsets, the first subset is the node itself and the set of neighboring nodes that are spatially closer to the center of gravity of the entire skeleton than the node itself, and the second subset is the set of neighboring nodes that are farther away from the center of gravity of the skeleton, thereby generating two adjacency matrices; Step S2-1-3: normalize the two generated adjacency matrices using the inverse of the square root of the degree, where the normalization formula is as follows: Where A is the adjacency matrix of the graph; D is the degree matrix of the graph, which is a diagonal matrix; is the square root of the inverse matrix of the degree matrix D; The normalized adjacency matrix obtained in this way is normalized A Making the sum of the adjacency weights of each node equal to 1 helps better capture the relationship between nodes in graph convolutional networks. This process is called symmetric normalization. The specific method of adding the DropBlockSke layer to obtain the optimized feature vector result is: Apply a binary mask to the input tensor to randomly drop features in certain areas to help the model prevent overfitting. The formula for DropBlockSke is as follows: M seed =casr(random.u(tf.shape(input_abs))<input_abs_norm×gamma) (3-1); M=matmul(M seed ,A) (3-2); mask=expand_dims(expand_dims(1-M,axis=0),axis=0) (4); In formula (2), α is a constant related to the key point num_point. The number of key points num_point in the present invention is 25, so α in the present invention is 1.92; keep_prob represents the probability of retaining the node; In formula (3-1), input_abs is the input feature vector of the layer, which represents the features or attributes of the nodes in the graph; input_abs_norm is the normalized result of input_abs, which is used to ensure that the values of different features or nodes are in the same range; gamma is a parameter that controls sparsity. The parameter types calculated by formula (2) are all float32, and finally a two-dimensional mask matrix M is generated. seed ; In formula (3-2), M_seed is the sparse mask matrix generated by formula (3-1), matmul is the matrix multiplication operation; M represents M seed The result of multiplying the matrix A is to expand M_seed to the graph structure; In formula (4), 1-M means changing 0 in M to 1 and 1 to 0 to generate a mask matrix; expand_dims(axis=0) means adding a dimension to the specified dimension; mask is the final mask matrix used to mask the input data; In formula (5), inputs represents the input data; size(mask) represents the total number of elements in the mask; reduce_sum(mask) represents the total number of non-zero elements in the mask; output is the final output of the DropBlockSke layer, which is achieved by multiplying the input data with the mask to achieve block dropping; The skeleton prediction network is composed of LSTM stacked twice, including two LSTM layers, Dropout layer and TimeDistributed layer. First, the output Y of the spatiotemporal graph convolutional network STGCN is converted into l As input, it is reshaped into a two-dimensional shape through the Reshape layer, i.e. (-1, x.shape[2]*x.shape[3]), integrating the time step and key point information; then, the data passes through an LSTM layer, and the Dropout layer is applied to prevent overfitting; then, the TimeDistributes layer is connected to replicate the information on the time dimension, so that the model can better process sequence data; then, after the data is reshaped again, it enters the second LSTM layer to further extract the timing information; similarly, after applying the Dropout and TimeDistributed layers again, the data is connected to the fully connected layer to map to the output space; finally, the coordinate prediction of the key point is generated; this structure is designed to process key point time series data, learn the timing information in the network and generate predictions for the key point positions; Step S3: Use the human posture dataset to train the improved STGCN model until the improved STGCN model reaches a low error or high accuracy index; Step S4: Use the trained network to evaluate the human posture video and obtain the predicted value of the corresponding evaluation score.
2. The method for human posture video assessment based on spatiotemporal graph convolutional network according to claim 1, characterized in that: The preprocessing in step S1 is as follows: the data is divided into multiple batches, and each batch contains a four-dimensional feature vector N*T*V*C of 100 time steps, where N represents the size of each batch, T represents the time series length of each sample, V represents the number of key points, and C represents the number of feature channels.
3. The method for human posture video assessment based on spatiotemporal graph convolutional network according to claim 1, characterized in that: The specific steps of performing time series prediction of key points in step S2 are: The output of the added skeleton prediction network is used as the input of the self-supervised loss function to form a new loss function This loss function It is composed of Huber function and MSE function, minimizing this loss function to optimize the spatiotemporal feature extraction network; The loss function The formula is as follows: in, In formula (6), Encoded i Indicates the predicted key point coordinate value of the current time step, seq_input_lt i Indicates the real key point coordinate value; L (mse) represents the mean square error, which is used here as the self-supervised loss function; In formula (7), L huber Represents the Huber loss function, which is the loss function used in traditional model evaluation; represents the predicted value of the evaluation score; h i represents the true value of the evaluation score, which comes from the score value of the subject in the human posture dataset constructed in step S1; δ is a threshold used to control the behavior of the loss function.
4. The method for human posture video assessment based on spatiotemporal graph convolutional network according to claim 3, characterized in that: In the process of training the improved STGCN model in step S3, the training process of the spatiotemporal feature extraction network part and the action assessment head network part are both trained using the loss function To express: