A taijiquan full-body posture estimation method, device and medium based on full convolution

Abstract

The application discloses a Taijiquan full-body posture estimation method and device based on full convolution, and a medium. Video data is acquired, and a single-frame image feature is extracted by preprocessing the video. Full-body posture estimation is performed based on the extracted feature to obtain a human skeleton key point heat map containing a hand and a head, and a human posture skeleton map is determined. A multi-frame time feature extraction and human posture estimation model is constructed, multi-feature posture estimation is performed according to the human posture skeleton map, multi-feature estimation results are fused, and a human posture estimation result is obtained. The human posture details of the hand and the head are fused to obtain a more accurate full-body posture estimation result. The multi-features of human key points are extracted and fused, and posture prediction is combined with time features, so that the precision and stability of the algorithm are enhanced, the multi-scale feature extraction of the video signal and the time domain feature of the signal itself are taken into account, the time and space features of the signal are considered, and the Taijiquan action posture estimation is more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to a Taijiquan full-body posture estimation method and device based on full convolution and a medium. BACKGROUND

[0002] Taijiquan is a traditional Chinese martial art that can strengthen the body, cultivate the mind and improve health. In the process of rehabilitation treatment through Taijiquan, there may be problems such as action errors and poor action coordination, which need to be analyzed and guided. With the application of deep learning and machine learning, there are methods for predicting human key points and estimating postures based on images or videos.

[0003] Chen et al. use a convolutional residual network to extract features from human motion videos, fuse the current frame image features with the previous frame feature maps, and determine the target human posture according to the multi-frame feature fusion features;

[0004] Jian et al. add a pyramid pooling module to the original convolutional feature extraction module to extract multi-scale features for posture estimation, thereby improving the accuracy of posture estimation.

[0005] Xiao et al. extract the target in each sample image and obtain a sample subgraph through linear interpolation, set anchor points in the subgraph, and use a neural network predictor to obtain the features of each anchor point relative to the target key points and realize posture estimation;

[0006] Yang et al. use an HRnet backbone network to extract human posture dynamic change features in multiple frames of images, and fuse and match the current frame prediction results to obtain multi-frame posture estimation results.

[0007] Defects of the prior art:

[0008] 1. The existing human posture estimation methods only through recurrent networks or machine learning methods often ignore the dynamic information such as speed and acceleration of human key points, which is necessary for identifying complex actions such as Taijiquan;

[0009] 2. The existing methods for feature extraction through convolution combined with recurrent networks weaken or ignore high-precision feature extraction of detailed parts such as hands and heads.

[0010] Taijiquan practice movements contain complex actions such as head rotation and hand gesture changes, and existing posture estimation methods usually only consider limb key points, ignoring the feature information of detailed parts. SUMMARY

[0011] The technical problem to be solved by the present application is that the existing posture estimation method usually only considers the limb key points, ignores high-precision feature extraction of detailed parts such as hands and heads of the human body, and aims to provide a Tai Chi whole-body posture estimation method, device and medium based on full convolution, which estimates the whole-body posture of a single frame image, fuses the human posture details of hands and heads through a multi-scale feature extraction method to obtain more accurate whole-body posture estimation results, extracts and fuses multi-features of human key points through a multi-frame time feature extraction and human posture estimation network, combines time features for posture prediction, and enhances the accuracy and stability of the algorithm.

[0012] The high-precision feature extraction of dynamic features and body parts is considered, and the accuracy of posture estimation is improved.

[0013] The present application is realized by the following technical solutions:

[0014] The present application provides a Tai Chi whole-body posture estimation method based on full convolution in the first aspect, which comprises the following specific steps:

[0015] Obtain video data, pre-process the video, and extract single frame image features;

[0016] Perform whole-body posture estimation based on the extracted features to obtain a human skeleton key point heat map containing hands and heads, and determine a human posture skeleton graph;

[0017] Construct a multi-frame time feature extraction and human posture estimation model, perform multi-feature posture estimation according to the human posture skeleton graph, fuse the multi-feature estimation results, and obtain a human posture estimation result.

[0018] The present application estimates the whole-body posture of a single frame image, fuses the human posture details of hands and heads through a multi-scale feature extraction method to obtain more accurate whole-body posture estimation results, extracts and fuses multi-features of human key points through a multi-frame time feature extraction and human posture estimation network, combines time features for posture prediction, enhances the accuracy and stability of the algorithm, considers the multi-scale feature extraction of video signals and the time domain features of the signals, that is, extracts and considers the time and space features of the signals, and makes the Tai Chi action posture estimation more accurate.

[0019] Further, the pre-processing of the video specifically comprises:

[0020] Obtain a human image I of the t-th frame of the video t , construct a ResNet-50 image feature extraction model, perform image basic feature extraction on I t , and obtain a feature map wherein w represents the width of the feature map set, h represents the height, and n represents the number of channels.

[0021] Further, the obtained human skeleton key point heat map containing hands and head specifically includes:

[0022] Constructing an HRNet model, and obtaining a full-body single-frame image feature set Performing body posture estimation, and outputting a human skeleton key point heat map;

[0023] Obtaining a head and hand bounding box division result according to the key point heat map;

[0024] According to the head and hand bounding box division result, combining Obtaining original image features of the head and hands, performing head and hand posture estimation on the original image features of the head and hands by using the HRNet model, and obtaining a human skeleton key point heat map of the hands and head.

[0025] Further, the determining human posture skeleton map Specifically includes:

[0026] Filling the heat map containing hand and head key points into the hand and head bounding box of the full-body key point heat map, and outputting a key point heat map containing full-body key points;

[0027] According to the probability of the key points in the heat map at each pixel point, selecting the pixel point with the highest probability as the output key point, and obtaining a human posture skeleton map Wherein, L is the number of video frames, N is the total number of estimated key points of the whole body, and D represents the dimension of each key point.

[0028] Further, the constructing multi-frame time feature extraction and human posture estimation model specifically includes:

[0029] Constructing a baseline branch for processing position feature information of key points;

[0030] Constructing a speed branch for processing speed feature information of key points;

[0031] Constructing an acceleration branch for processing acceleration feature information of key points.

[0032] Further, the obtaining human posture estimation result specifically includes:

[0033] Inputting the human posture skeleton map Into the multi-frame time feature extraction and human posture estimation model for convolution operation;

[0034] Performing time feature extraction on the data after convolution by using a sliding window method until all video frames are input, and obtaining a position prediction result of the human posture, a speed prediction result of the human posture and an acceleration prediction result of the human posture;

[0035] The position prediction result, the speed prediction result and the acceleration prediction result of the human body posture are fused to obtain a human body posture estimation result.

[0036] Further, the time feature extraction by using the sliding window method specifically comprises:

[0037] The size of the sliding window is set as The first input is the first frame to the Tth frame, the second input is the first+t frame to the T+t frame, and so on until the Lth frame is input, wherein T represents the frame number.

[0038] Further, the method further comprises training the multi-frame time feature extraction and human body posture estimation model:

[0039] The real human body posture key point distribution is obtained, the human body posture estimation result obtained after multi-frame feature extraction is combined, and a position loss function is determined;

[0040] The human body posture acceleration and the acceleration calculated according to the predicted posture are obtained, and an acceleration loss function is determined;

[0041] The loss weight parameter is set, the position loss function and the acceleration loss function are combined to determine a loss function;

[0042] Based on the loss function, the gradient descent method is used to train the multi-frame time feature extraction and human body posture estimation model.

[0043] The third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize a Taijiquan full-body posture estimation method based on full convolution.

[0044] The third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to realize a Taijiquan full-body posture estimation method based on full convolution.

[0045] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0046] By performing full-body posture estimation on a single frame image, by using a multi-scale feature extraction method, the human body posture details of the hand and head are fused to obtain more accurate full-body posture estimation results, by constructing a multi-frame time feature extraction and human body posture estimation network, the multi-features of the human body key points are extracted and fused, and the posture is predicted in combination with the time features, the accuracy and stability of the algorithm are enhanced, the multi-scale feature extraction of the video signal and the time domain features of the signal itself are taken into account, that is, the time and space features of the signal are extracted and considered, so that the Taijiquan action posture estimation is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0047] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0048] Figure 1 This is an overall flowchart of an embodiment of the present invention;

[0049] Figure 2 This is a multi-frame temporal feature extraction and human pose estimation model in this embodiment of the invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0051] Example 1

[0052] like Figure 1 As shown, the first aspect of this embodiment provides a method for estimating the whole-body posture of Tai Chi Chuan based on fully convolution, including the following specific steps:

[0053] Acquire video data, preprocess the video, and extract single-frame image features;

[0054] Based on the extracted features, full-body pose estimation is performed to obtain a heatmap of key points of the human skeleton including the hands and head, and the human pose skeleton map is determined.

[0055] A multi-frame temporal feature extraction and human pose estimation model is constructed. Multi-feature pose estimation is performed based on the human pose skeleton map. The multi-feature estimation results are then fused to obtain the human pose estimation result.

[0056] This embodiment estimates the whole-body pose by performing full-body pose estimation on a single frame image. It uses a multi-scale feature extraction method to fuse human pose details of the hands and head to obtain a more accurate full-body pose estimation result. By constructing a multi-frame temporal feature extraction and human pose estimation network, it extracts and fuses multiple features of human key points and combines temporal features for pose prediction, which enhances the accuracy and stability of the algorithm. It takes into account both the multi-scale feature extraction of the video signal and the temporal features of the signal itself, that is, it extracts and considers the temporal and spatial features of the signal, making the Tai Chi movement pose estimation more accurate.

[0057] In some possible embodiments, video preprocessing specifically includes:

[0058] obtain a human body image I of the t-th frame of the video t , construct a ResNet-50 image feature extraction model, perform image basic feature extraction on I t , and obtain a feature map wherein w represents the width of the feature map set, h represents the height, and n represents the number of channels.

[0059] In some possible embodiments, obtaining the human body skeleton key point heat map containing the hand and the head specifically includes:

[0060] constructing an HRNet model, performing body posture estimation on the full-body single-frame image feature set , and outputting a human body skeleton key point heat map;

[0061] obtaining a head and hand bounding box division result according to the key point heat map;

[0062] combining the head and hand bounding box division result with obtaining original image features of the head and the hand, performing head and hand posture estimation on the original image features of the head and the hand by using the HRNet model, and obtaining a human body skeleton key point heat map of the head and the hand.

[0063] In some possible embodiments, determining the human body posture skeleton map specifically includes:

[0064] filling the heat map containing the hand and the head key points into the hand and head bounding box of the full-body key point heat map, and outputting a key point heat map containing full-body key points;

[0065] selecting, according to the probability of the key points in the heat map at each pixel point, the pixel point with the highest probability as the output key point, and obtaining the human body posture skeleton map wherein L is the number of video frames, N is the total number of estimated full-body key points, and D represents the dimension of each key point.

[0066] In some possible embodiments, as shown in Figure 2 , the left image is a whole model diagram of a multi-frame time feature extraction and human body posture estimation model, the first branch from left to right is a baseline branch for processing position feature information of key points, the second branch is a speed branch for processing speed feature information of key points, and the third branch is an acceleration branch for processing acceleration feature information of key points. The right image is a residual connection block in M1 / 2 / 3Blocks, which can be stacked for multiple layers, and in this embodiment, the residual connection block is set to 4 layers;

[0067] Wherein, the FC Layer is a full convolutional neural network module, the LeakyReLu is a Relu activation layer, the Fusion Layer is a full connection neural network for data fusion, and the represents matrix addition, represents matrix subtraction.

[0068] The human pose skeleton graph is input into a multi-frame time feature extraction and human pose estimation model, and specifically includes:

[0069] Input [C, T, 2], C represents the number of key points, T represents the number of frames input at a time, and 2 represents 2D coordinates, wherein C is equal to the total number of key points N output in the previous step, T is less than the total number of video frames L, and a sliding window method is used to extract time features every time T adjacent frames are input, and the size of the sliding window is That is, the first time the first frame to the Tth frame is input, the second time the 1+t frame to the T+t frame is input, and so on until all L frames are input.

[0070] The size of the FC layer convolution kernel is 3*3, k represents the number of convolution kernels of the first FC layer in the branch, the dimension of [C, T, 2] becomes [C, T, k] after passing through the first FC layer, the M1 / 2 / 3Blocks residual module does not change the feature dimension of the input and output, e represents the number of convolution kernels of the second FC layer in the branch, the dimension of [C, T, k] becomes [C, T, e] after passing through the corresponding convolution layer, and the other two branches are the same, the three branches respectively output the position prediction result of the human pose, the speed prediction result of the human pose and the acceleration prediction result of the human pose, and the output results of the three branches are fused through the full connection data fusion layer to obtain the human pose estimation result.

[0071] The specific calculation steps include: on the ith key point in the tth frame, the position prediction of the human pose is the speed and the acceleration can be expressed as:

[0072]

[0073]

[0074] Wherein s is the time difference between adjacent two frames (the value of the present application is 1), j is taken as 1, then the formula is simplified as:

[0075]

[0076]

[0077] As shown in the network model, the first branch from left to right is the baseline branch, processing the position error of key points, the second branch is the speed branch, and the third branch is the acceleration branch.

[0078] The overall model can be represented as the following function:

[0079]

[0080] wherein, LF(·) represents linear fusion, FConv(·) represents a full convolutional layer branch, is the human pose estimation result obtained after multi-frame feature extraction, i represents the ith key point, t represents the tth frame, that is, the specific distribution of the human skeleton key points in each frame of the pixel graph, which can also be called a taijiquan human motion sequence.

[0081] In some possible embodiments, the multi-frame time feature extraction and human pose estimation model is also trained:

[0082] The real human pose key point distribution is obtained, combined with the human pose estimation result obtained after multi-frame feature extraction, to determine the position loss function;

[0083] The human pose acceleration and the acceleration calculated according to the predicted pose are obtained to determine the acceleration loss function;

[0084] The loss weight parameter is set, and the loss function is determined in combination with the position loss function and the acceleration loss function;

[0085] Based on the loss function, the multi-frame time feature extraction and human pose estimation model is trained using the gradient descent method;

[0086] The specific calculation steps include:

[0087]

[0088]

[0089] L final =L pose +αL acc

[0090] wherein, Y i,t is the real human pose key point distribution, is the acceleration calculated according to the predicted pose, A i,t is the human pose acceleration, and a is the loss weight parameter. The model is trained using the gradient descent method.

[0091] The third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements a Tai Chi whole body posture estimation method based on full convolution when executing the program.

[0092] The third aspect of the present embodiment provides a computer readable storage medium having a computer program stored thereon, and the program implements a Tai Chi whole body posture estimation method based on full convolution when executed by a processor.

[0093] The above specific embodiments further explain the purposes, technical solutions and advantages of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A full convolution-based Tai Chi whole body posture estimation method, characterized in that, The method comprises the following specific steps: Obtaining video data, pre-processing the video, and extracting single-frame image features; Based on the extracted features, whole body posture estimation is performed to obtain a human skeleton key point heat map containing a hand and a head, and a human posture skeleton graph is determined; A multi-frame time feature extraction and human posture estimation model is constructed, multi-feature posture estimation is performed according to the human posture skeleton graph, the multi-feature estimation results are fused, and human posture estimation results are obtained; The construction of the multi-frame time feature extraction and human posture estimation model specifically includes: constructing a baseline branch for processing position feature information of key points; constructing a speed branch for processing speed feature information of key points; and constructing an acceleration branch for processing acceleration feature information of key points; The human pose estimation result specifically comprises: inputting the human pose skeleton graph into a multi-frame time feature extraction and human pose estimation model for convolution operation; performing time feature extraction on the data after convolution by using a sliding window method until all video frames are input, to obtain a human pose position prediction result, a human pose speed prediction result and a human pose acceleration prediction result; and fusing the human pose position prediction result, the human pose speed prediction result and the human pose acceleration prediction result to obtain a human pose estimation result. 2.The Tai Chi full-body posture estimation method based on full convolution according to claim 1, wherein, The pre-processing of the video specifically includes: Acquire human body image of the t-th frame of video , construct a ResNet-50 image feature extraction model, perform image basic feature extraction on , and obtain a feature map , wherein w represents the width of the feature map set, h represents the height, and n represents the number of channels. 3.The Tai Chi full-body posture estimation method based on full convolution according to claim 2, characterized in that, The human skeleton key point heat map containing the hand and the head specifically includes: Constructing an HRNet model to obtain a feature set of a single frame of a whole body Performing body posture estimation to output a human skeleton key point heat map; A head and hand bounding box division result is obtained according to the key point heat map; According to the head and hand bounding box division result, combined with Obtain the original image features of the head and hand, use the HRNet model to estimate the head and hand posture of the original image features of the head and hand, and obtain the human skeleton key point heat map of the hand and head.

4. The full convolution-based Tai Chi whole body posture estimation method according to claim 3, characterized in that, The determining human pose skeleton map Specifically comprises: The key point heat map containing the hand and the head key points is filled into the hand and head bounding boxes of the whole body key point heat map, and a key point heat map containing whole body key points is outputted; According to the probability of the key point in each pixel point in the heat map, the pixel point with the highest probability is selected as the output key point, and a human body posture skeleton graph is obtained wherein L is the number of video frames, N is the total number of body estimation key points, and D represents the dimension of each key point.

5. The full convolution-based Tai Chi whole body posture estimation method according to claim 1, characterized in that, The time feature extraction by using the sliding window method specifically includes: The size of the sliding window is set as , the first time input the first frame to the T frame, the second time input the first + t frame to the T + t frame, until the video frame number L frame is input, wherein T is the frame number.

6. The full convolution-based Tai Chi whole body posture estimation method according to claim 1, characterized in that, The multi-frame time feature extraction and human posture estimation model is also trained: Real human posture key point distribution is obtained, and a position loss function is determined in combination with human posture estimation results obtained after multi-frame feature extraction; Human posture acceleration and acceleration calculated according to the predicted posture are obtained, and an acceleration loss function is determined; A loss weight parameter is set, and a loss function is determined in combination with the position loss function and the acceleration loss function; Based on the loss function, the multi-frame time feature extraction and human posture estimation model is trained by using a gradient descent method.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the whole convolution-based Taijiquan whole body posture estimation method according to any one of claims 1 to 6. 8.A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to realize the whole convolution-based Taijiquan whole body posture estimation method according to any one of claims 1 to 6.

Images