An end-to-end keypoint and convolutional neural network based action recognition method

Abstract

The application discloses an end-to-end action recognition method based on key points and a convolutional neural network, which comprises the following steps: (1) sampling from a time sequence video to obtain a plurality of time sequence picture sequences; (2) selecting a neural network at an edge device end as a key point detection model to predict a plurality of key point data from the plurality of time sequence picture sequences; (3) using a key point prediction large model to distill the key point detection model, or using a real label to train the key point detection model; (4) generating a Gaussian distribution heat map according to the predicted key points; (5) extracting a space-time feature on the plurality of Gaussian distribution heat maps; (6) predicting an action type according to the space-time feature; and (7) setting a target function to train a model, and deploying the trained key point detection model and action recognition model at the edge device end and a cloud end respectively to perform real-time target action detection. By using the application, the accuracy and authenticity of the action recognition based on key points can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of action recognition, and particularly relates to an end-to-end action recognition method based on key points and a convolutional neural network. BACKGROUND

[0002] With the release of a large number of public video data sets, the field of video understanding and action recognition has attracted extensive attention from the industry and academia. For example, the action recognition product of SENSETIME is applied to the fight recognition of public security and schools, the fall recognition of nursing homes, the fall recognition of escalators in shopping malls and other scenes.

[0003] There are two mainstream methods in the field of action recognition. One is RGB-based action recognition, and the other is key point-based action recognition. These two methods have the following advantages and disadvantages: 1. Generally speaking, the RGB-based method has higher recognition accuracy, especially in recent years, the self-attention-based method has brought the recognition accuracy of the RGB-based method to a new height; 2. The RGB style has less impact on the key point-based recognition, so the key point-based recognition model has better performance in the case of few-shot fine-tuning or zero-shot; 3. The key point-based action recognition method is a method that combines data desensitization and strong recognition explainability, which can effectively protect the privacy data of users and provide explainability basis for identification of false positives and false negatives; 4. The key point-based action recognition method can deploy the key point recognition model and the action recognition model on the edge device and the cloud respectively, which can effectively reduce the server pressure of the edge device, and the network bandwidth usage is one hundred thousandth of the RGB recognition model deployed in the cloud.

[0004] At present, the mainstream key point-based action recognition technology is divided into two types. One is the graph convolutional neural network represented by the ST-GCN (Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition) proposed by Joao Carreira et al. in the 2018 AAAI conference, which builds a graph according to the position information of the key points in space and time, and then applies graph convolution on the graph to extract features and classify and recognize. The other is the three-dimensional convolutional neural network represented by PoseConv3D (Revisiting Skeleton-based Action Recognition) proposed by Haodong Duan et al. in the 2022 CVRP conference. This method first spreads a heat map around each key point, then stacks the heat maps into a high-dimensional heat map, and finally applies three-dimensional convolution to the multi-frame high-dimensional heat map to extract features and classify and recognize.

[0005] Human keypoint action recognition still needs a preposed human keypoint extractor, and the current mainstream keypoint recognition technology is divided into top-down and bottom-up. The top-down method first uses target detection to frame the human body, and then extracts key points from the framed human body graph. For example, Ke Sun et al. proposed HRNet in 2019 CVPR conference Deep High-Resolution Representation Learning for Human Pose Estimation, which is a top-down key point extraction deep convolutional neural network. HRNet increases the accuracy of key point recognition by fusing image features at multiple resolutions from high to low. Recently, there are also top-down large models based on self-attention. Yufei Xu proposed ViTPose in 2022 Neurips ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation, which uses a visual Transformer as the backbone neural network for feature extraction. The number of parameters can reach 1B, and the accuracy on multiple datasets is significantly higher than that of deep convolutional neural networks. The bottom-up method first extracts the corresponding key points according to the preset number of people, and then matches the corresponding key points for each human body. For example, Zhe Cao et al. proposed OpenPose in OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, which calculates the affinity field between key points after extracting all key points, and finally matches each key point to the corresponding human body according to the affinity field.

[0006] However, the current mainstream keypoint-based action recognition method has a very big disadvantage compared to the RGB-based method in the industrial landing scene, mainly in the engineering difficulty and training difficulty brought by the two-stage training and the more expensive keypoint data cost compared to RGB. Secondly, using labeled data for training and testing will have a very big gap with the actual deployment training and testing accuracy, because the key points of the keypoint-based action recognition model do not come from the key point detection model that will be actually deployed, but from the labeled or pre-predicted stored key points. Both pre-stored or labeled will bring engineering overhead. When weighing the pros and cons of keypoint-based action recognition models and RGB-based models, industry products and researchers often give up keypoint models because of the additional overhead of keypoint models compared to RGB models, even if they have many benefits.

[0007] Although the above said many disadvantages of key point based action recognition compared to RGB based action recognition, in actual products and businesses, there is still a very big demand for this method of key point based action recognition, in actual business, customers usually do not like or do not want to upload local video, photo and other private data to the server of the algorithm service provider, in such occasions, key point based action detection can be used as a technology that can well complete the action detection demand under the condition of guaranteeing user privacy. SUMMARY

[0008] The application provides an end-to-end key point and convolutional neural network based action recognition method, so that researchers do not need to train key point detection network and key point based action recognition network separately, and the accuracy and authenticity of the test accuracy and deployment accuracy of the end-to-end test are aligned, improving the test accuracy and authenticity of the key point based action recognition.

[0009] An end-to-end key point and convolutional neural network based action recognition method, comprising:

[0010] (1) sampling from a time sequence video to obtain a plurality of frame time sequence picture sequences;

[0011] (2) selecting a deep convolutional neural network at the edge device end as a key point detection model to predict a plurality of frame human key point data from the plurality of frame time sequence picture sequences;

[0012] (3) using a key point prediction large model to distill the key point detection model, or using real labels to train the key point detection model in step (2) in the case of real labels;

[0013] (4) generating a key point centered Gaussian distribution heat map according to the key points predicted in step (2);

[0014] (5) performing three-dimensional convolution on the plurality of frame Gaussian distribution heat maps to extract spatio-temporal features;

[0015] (6) inputting the extracted spatio-temporal features into an action recognition model to predict an action type;

[0016] (7) setting an objective function to train the key point detection model and the action recognition model, and deploying the trained key point detection model and the action recognition model at the edge device end and the cloud end respectively to perform real-time target action detection.

[0017] In step (1), a uniform sampler is used to divide an N frame time sequence video into T video segments with equal length, and then a frame of data is randomly extracted from each video segment to obtain T frame RGB pictures.

[0018] In step (2), the goal of the key point detection model is to detect key points from a 3×H×W RGB image I. t The predicted heatmaps of K H'×W' {H1,H2,…,H} are obtained. K}, each heatmap H k This represents the probability distribution of the k-th key point on this map. The coordinates with the highest probability are selected from the distribution map as the predicted coordinates of the key point.

[0019] In step (3), the key point prediction big model adopts the visual Transformer big model; if the labels are available, the real labels are used; otherwise, the visual Transformer big model is used to predict the corresponding K key points for the images extracted from the video. The purpose is to maintain end-to-end training in this step even without real labels.

[0020] In step (4), the generation of Gaussian distribution heatmaps is divided into two types: joint heatmap generation and limb heatmap generation. The formula for generating joint heatmaps is as follows:

[0021]

[0022] In the formula, the triple (x k ,y k ,c k The coordinates and confidence score of the k-th joint from the keypoint detection model are represented by the heatmap H. k The coordinates with the highest confidence are used to generate a joint heatmap centered on the joints using a Gaussian distribution; σ is the variance of the Gaussian distribution.

[0023] The formula for generating a limb thermogram is as follows:

[0024]

[0025] In the formula, the k-th limb is the a-th limb. k The joint and the bth k The lines connecting the joints, These are the confidence scores for the two joints, where D represents the distance from coordinate (i,j) to coordinate (a). k ,b k The distance calculation function is used to generate a heatmap centered on limb segments using a Gaussian distribution, where σ is the variance of the Gaussian distribution.

[0026] In step (5), the three-dimensional convolution is performed by sliding a 3D convolution kernel of size N×N×N onto each pixel of the input data. The convolution is applied to the key point heatmap, and the convolution sliding dimension is T×H×W, which integrates the spatial and temporal information in the video. The K-dimensional heatmap generated by the joint is stacked as the initial channel feature.

[0027] In step (6), the extracted spatio-temporal features are input into a fully connected classifier, and classification log values are obtained through a softmax activation layer.

[0028] In step (7), errors are respectively calculated according to the classification results of the action recognition model and the heat map of the key point detection model, and network parameters are optimized through back propagation. classifier As follows:

[0029] L classifier (p,q)=-∑ i p i log(q i )

[0030] The loss of classification is calculated using the cross-entropy formula, where p represents the real probability distribution, q represents the probability distribution predicted by the model, log represents the logarithm with base 2, and i represents the index of the class; the smaller the value of cross-entropy, the smaller the difference between the two probability distributions, and the closer the prediction result of the model to the real result.

[0031] The error L keypoint of the key point detection model is as follows:

[0032] L keypoint =MSE(H predicted ,H ground truth )

[0033] where H predicted is the heat map of a joint predicted by the key point detection model, H ground truth is a Gaussian distribution heat map centered on the real joint coordinates, and MSE is the pixel-wise mean square error.

[0034] Compared with the prior art, the present application has the following beneficial effects:

[0035] 1. Traditional key point-based action recognition requires training of key point recognition neural networks and action recognition neural networks respectively. However, the present application realizes end-to-end key point-based action recognition, which can complete the entire process only once.

[0036] 2. The previous key point-based action recognition method directly uses labeled key points as input for the action recognition network, which makes it difficult to maintain consistency between the accuracy during training and the accuracy during actual deployment. In contrast, the present application realizes end-to-end training, which makes the accuracy during training perfectly aligned with the accuracy during deployment. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is the overall framework design diagram of the method of the present application;

[0038] Figure 2 is a system flowchart designed according to the method of the present application;

[0039] Figure 3 is a multi-stage multi-resolution subnetwork structure of an edge device end key point detection model in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The present application will be further described in detail below in conjunction with the drawings and embodiments, and it should be noted that the following embodiments are intended to facilitate the understanding of the present application and do not limit the present application in any way.

[0041] The present application provides an end-to-end key point and convolutional neural network-based action recognition method, which does not need to train a key point detection model and a key point-based action recognition model separately, and aligns the precision of end-to-end testing and deployment precision (the previous key point-based action detection method directly uses the precision of labeled key point testing, not the actual end-to-end deployment precision), thereby improving the accuracy and authenticity of the test precision of the key point-based action recognition. In addition, a training method that only requires a video dataset without key point labeling is derived: a Transformer large model is used as a teacher model to train an edge device end key point detection small model in a knowledge distillation manner, which greatly reduces the overhead caused by data preparation.

[0042] The reason why an edge device end key point detection model is still needed instead of directly using a large model as a key point detection model is that the key role of the key point-based action recognition itself is privacy protection, and it is difficult to deploy a Transformer large model on an edge device end device in terms of performance and adaptation. Therefore, an edge device end small model needs to be selected as the actual deployment key point detection model.

[0043] An end-to-end key point and convolutional neural network-based action recognition method, when there are labeled key points, uses labeled key points for training, and if there are no labeled key points, uses a visual transformer key point detection large model for knowledge distillation, including the following steps:

[0044] S01, sampling from a time sequence video to obtain a plurality of frame time sequence pictures.

[0045] In the action recognition process, since the video is time sequence data, a fixed T frame RGB picture needs to be selected from a N frame video to input the network, and the sampler is a uniform sampler. The frame extractor divides the N frame video into T video segments with equal length, and then randomly extracts one frame of data from each segment. The N frame picture {I1, I N ,…,I NThe T-frame RGB picture Tx3xHxW is sampled from the video, and usually, the action recognition process reduces the redundancy of data information through this sampling.

[0046] S02, a key point prediction deep convolutional neural network at the edge device end is used to predict multiple frames of human key point data from a multiple-frame time sequence picture sequence.

[0047] The purpose of the key point detection model is to predict K heat maps {H1, H2,..., H t from a 3xHxW RGB picture I K , and each heat map H k represents the distribution probability of the kth key point on the map. The coordinate with the maximum probability is selected as the key point prediction coordinate in the distribution map. The deep convolutional neural network at the edge device end is selected as the key point detection model, and the visual transformer large model is selected as the key point prediction large model in step S03.

[0048] S03, the key point prediction large model is used to distill the key point prediction model at the edge device end, or the real label is used to train the key point detection model in step S02.

[0049] If the label is available, the real label is used, otherwise the visual transformer large model is used to predict the corresponding K key points for the picture extracted from the video, so that end-to-end training can be maintained in this step without real labels.

[0050] S04, a Gaussian distribution heat map centered on the predicted key point is generated.

[0051] The label key point or the key point predicted by the real label or the key point prediction large model in step S03 is used during training, and the key point predicted by the key point detection model at the edge device end in step S02 is used during inference and testing. The Gaussian distribution heat map centered on the key point is generated, and then the heat map tensor size is scaled to a fixed value T x K x H' x W', data augmentation includes random cropping, random rescaling, key point random symmetric flipping, etc., and data processing includes key point heat map generation and human body frame cropping. Data augmentation is only applied in training and not in inference testing.

[0052] The heat map generation includes joint heat map generation and limb heat map generation, and the generation formula of the joint heat map is as follows:

[0053]

[0054] In the formula, the triplets (x k , yk ,c k The coordinates and confidence score of the k-th joint from the pose detection model are represented by the heatmap H. k The coordinates with the highest confidence are used in this invention to generate joint heatmaps centered on the joints using a Gaussian distribution. σ represents the variance of the Gaussian distribution.

[0055] The formula for generating a limb thermogram is as follows:

[0056]

[0057] In the formula, the k-th limb is the a-th limb. k The joint and the bth k The lines connecting the joints, These are the confidence scores for the two joints, where D represents the distance from coordinate (i,j) to coordinate (a). k ,b k The distance calculation function is used in this invention to generate a heatmap centered on limb line segments using a Gaussian distribution. σ is the variance of the Gaussian distribution.

[0058] S05 applies 3D convolution to multiple frames of heatmaps to extract spatiotemporal features.

[0059] A 3D deep convolutional neural network is used as the backbone model to extract spatiotemporal features from multi-frame keypoint heatmap data. 3D convolution is a commonly used signal processing technique, typically used for the analysis and processing of multi-dimensional data such as images, audio, and video. 3D convolution is a linear filter that performs the convolution operation by sliding an N×N×N 3D convolution kernel (also called a filter) across each pixel of the input data. The convolution operates on the keypoint heatmap tensor described in step S04, with a sliding dimension of T×H×W, fusing spatial and temporal information from the video. The initial channel size is K, meaning the K dimensions of the stacked heatmaps generated by the joints are used as the initial channel features.

[0060] S06. Input the extracted spatiotemporal features into the action recognition model to predict the action type.

[0061] The extracted spatiotemporal features are input into a fully connected classifier, and the classification log value is obtained through a softmax activation layer.

[0062] S07. Train the model according to the objective function, and deploy the trained key point detection model and action recognition model on the edge device and the cloud respectively for real-time target detection.

[0063] Based on the classification results of the action recognition model and the heatmap of the keypoint predictor, the errors are calculated separately, and backpropagation is performed to optimize the network parameters, where the error L of the classifier is... classifier as follows:

[0064] L classifier (p,q)=-∑ i p i log(q i )

[0065] The loss of classification is calculated using the cross-entropy formula. Where p represents the real probability distribution, q represents the probability distribution predicted by the model, log represents the logarithm with base 2, and i represents the index of the class. The smaller the value of cross-entropy, the smaller the difference between the two probability distributions, and the closer the prediction result of the model to the real result. In machine learning, cross-entropy is often used as a loss function to optimize the parameters of the model.

[0066] The error L of the key point detection model keypoint is as follows:

[0067] L keypoint =MSE(H predicted ,H ground truth )

[0068] Where H predicted is the heat map of a joint predicted by the key point detection model, H ground truth is the Gaussian distribution heat map centered on the real joint coordinates, and MSE is the pixel-wise mean square error. In statistics and machine learning, mean square error is a commonly used measure method to measure the difference between the predicted value and the real value. Mean square error is the average of the square of each sample's prediction error, commonly used for evaluation of regression problems. The smaller the mean square error, the closer the model's prediction to the real value.

[0069] After the model training is completed, the key point detection model and the action recognition model are deployed on the edge device and the cloud device respectively to perform real-time action recognition.

[0070] As Figure 1 shown, the framework of the present application mainly includes three parts: (a) the key point detection model of the edge device, which inputs an image and outputs the key points of each human joint in the image. (b) the action recognition model based on key points, which inputs key points and outputs the action recognition result. (c) the key point prediction transformer large model, which uses its powerful zero-shot prediction ability to distill key point detection knowledge for the key point detection model of the edge device without key point label data. The specific steps are described as follows:

[0071] (a) The key point detection model of the edge device aims to predict K key points of the human body. The current mainstream method is to convert the problem into predicting K key point probability heat maps, and the pixel coordinates with the maximum probability in the heat map are taken as the predicted key point coordinates. The basic steps are as follows:

[0072] 1. In multi-stage convolution, multi-resolution sub-networks are used to extract multi-resolution feature maps. The neural network for pose estimation in this invention is constructed by connecting high-resolution sub-networks and low-resolution sub-networks together. Each sub-network constitutes a stage, consisting of a series of convolutional layers, and there is a down-sampling layer between adjacent sub-networks to halve the resolution.

[0073] 2. As shown in Figure 3 , this invention constructs parallel multi-resolution sub-networks {N11, N21, N22, N31, N32, N33}, where s is the index of the stage, r is the index of the resolution, and Nsr represents the sub-network of the s-th stage with the resolution of 2 -(1-r) of the original image.

[0074] 3. Information is exchanged among multiple resolutions through up-sampling and down-sampling. Each sub-network in a stage receives information from all other resolutions in the same stage, and this technique realizes the information fusion of multiple resolutions.

[0075] 4. Regression heat map calculates error. The last stage of high-resolution feature map will be directly used to regress the probability heat map. The sum of all pixel probabilities on the heat map is 1. Finally, the mean square error MSE(H predicted , H ground truth ) is calculated by subtracting the joint coordinates of the label or the joint coordinates predicted by the large model, and the loss of the deep learning neural network is used to update the convolutional parameters through backpropagation.

[0076] (b) Action recognition model based on key points, which aims to receive multi-frame key point data, and predict the action of the human body by extracting the temporal and spatial features of multi-frame key points. The basic steps are as follows:

[0077] 1. Construct a Gaussian distribution heat map centered on the key point coordinates. There are three ways to generate it. One is to generate only joint heat maps, one is to generate only limb heat maps, and one is to generate both joint heat maps and limb heat maps. Among the three, the effect is enhanced in turn, but the computational complexity is also increased in turn. Therefore, it is necessary to decide which heat map to use according to the actual scene. In the following, it will be referred to as key point heat map.

[0078] 2. Use a three-dimensional deep convolutional neural network to extract the spatio-temporal feature information of multi-frame key point data by moving in the long, wide, and time dimensions of the stacked heat map tensor.

[0079] 3. The obtained feature map is input into a fully connected neural network for classification. The fully connected neural network is trained and inferred according to the features extracted by the three-dimensional deep convolutional neural network to identify different actions. The output of the fully connected neural network represents the classification results of the action recognition task. Finally, the error is calculated using cross-entropy and the parameters are updated by backpropagation.

[0080] (c) Distill knowledge from the key point Transformer large model to the key point detection model on the edge device end without key point real labels. Query the cache with the training and testing video index as the key value to see if the large model has previously inferred results. If the cache is hit, use the previously inferred results as real label data to calculate the error of the small model. During training, use the results of the large model as the input of the action recognition predictor, and during testing, select the results of the key point small model as the input of the action recognition predictor. In this way, the quality of the training and the authenticity of the test results are guaranteed.

[0081] Figure 2 is a system flowchart designed according to the method of the present application. A total of five modules are included, namely a video sampler module, a key point predictor module, a key point large model distillation module, a key point data processing module, and a key point-based action predictor module.

[0082] The above-described embodiments have described the technical solutions and beneficial effects of the present application in detail. It should be understood that the above-described is only a specific embodiment of the present application and is not intended to limit the present application. Any modification, supplement and equivalent replacement made within the principle range of the present application should be included within the protection scope of the present application.

Claims

1. A method of end-to-end keypoint and convolutional neural network based action recognition, characterized in that, Comprise: (1) sample from the time sequence video, obtain a sequence of multiple frames of time sequence pictures; (2) select a deep convolutional neural network at the edge device end as a key point detection model, and predict multiple frames of human key point data from the sequence of multiple frames of time sequence pictures; (3) use a key point prediction large model to distill the key point detection model, or use real labels to train the key point detection model in step (2) under the condition of real labels; The key point prediction large model adopts a visual transformer large model; if labels are available, real labels are used, otherwise, the visual transformer large model is used to predict K key points corresponding to the pictures extracted from the video, the purpose being to maintain end-to-end training in this step without real labels; (4) generate a Gaussian distribution heat map centered on the key points according to the key points predicted in step (2); (5) perform three-dimensional convolution on the multiple frames of Gaussian distribution heat maps to extract spatio-temporal features; (6) input the extracted spatio-temporal features into an action recognition model to predict an action type; (7) set an objective function to train the key point detection model and the action recognition model, and deploy the trained key point detection model and the action recognition model on the edge device end and the cloud end respectively to perform real-time target action detection. 2.The end-to-end keypoint and convolutional neural network based action recognition method of claim 1, wherein, In step (1), a uniform sampler is used to divide an N-frame time sequence video into T video segments with equal length, and then a frame of data is randomly extracted from each video segment to obtain T frames of RGB pictures. 3.The end-to-end keypoint and convolutional neural network based action recognition method of claim 1, wherein, In step (2), the purpose of the key point detection model is to predict K heat maps {h1, h2, …, hK} of size H' x W' from an RGB image I of size 3 x H x W t , where each heat map h K k represents the probability distribution of the k-th key point on the map. The coordinate with the highest probability is selected as the predicted coordinate of the key point. k In step (2), the purpose of the key point detection model is to predict K heat maps {h1, h2, …, hK} of size H' x W' from an RGB image I of size 3 x H x W t , where each heat map h K k represents the probability distribution of the k-th key point on the map. The coordinate with the highest probability is selected as the predicted coordinate of the key point. k In step (2), the 4.The end-to-end keypoint and convolutional neural network based action recognition method of claim 1, wherein, In step (4), Gaussian distribution heat map generation includes joint heat map generation and limb heat map generation, and the generation formula of the joint heat map is as follows: In the formula, the triplet (x k ,y k ,c k ) is the coordinate and confidence of the kth joint from the key point detection model, that is, the coordinate with the highest confidence in the heat map H k ; the joint heat map centered on the joint is generated in a Gaussian distribution; and σ is the variance of the Gaussian distribution. The generation formula of the limb heat map is as follows: where the kth limb is the line connecting the ath joint and the bth joint, k k are the confidence of the two joints respectively, D is the distance calculation function from coordinate (i,j) to coordinate (a k ,b k ), the heat map centered on the limb line segment is generated by Gaussian distribution, and σ is the variance of Gaussian distribution.​​ 5.The end-to-end keypoint and convolutional neural network based action recognition method of claim 1, wherein, In step (5), three-dimensional convolution is realized by sliding a 3D convolution kernel with a size of N×N×N to each pixel of the input data; the convolution operates on the key point heat map, and the convolution sliding dimension is T×H×W, which fuses the spatial information and the temporal information in the video, and stacks the K dimensions of the generated joint heat map as the initial channel feature. 6.The end-to-end keypoint and convolutional neural network based action recognition method of claim 1, wherein, In step (6), the extracted spatio-temporal features are input into a fully connected classifier, and classification logarithmic values are obtained through a softmax activation layer.

7. The end-to-end keypoint and convolutional neural network based action recognition method of claim 1, wherein, In step (7), the network parameters are optimized by back propagation according to the error calculated respectively based on the classification result of the action recognition model and the heat map of the key point detection model, wherein the error L of the action recognition model is classifier As follows: L classifier (p,q) = -∑ i p i log(q i ) The loss of classification is calculated using the cross-entropy formula, wherein p represents the real probability distribution, q represents the probability distribution predicted by the model, log represents the logarithm with base 2, and i represents the index of the class; the smaller the value of cross-entropy, the smaller the difference between the two probability distributions, and the closer the prediction result of the model to the real result; Error L of the key point detection model keypoint As follows: L keyoint = MSE(H predicted , H groundtruth ) where H predicted is the predicted heatmap of a joint by the keypoint detection model, H groundtruth is the ground truth heatmap centered at the real joint coordinate, and MSE is the pixel-wise mean square error.

Images