Ba Duan Jin movement evaluation method and system based on dynamic programming and deep learning

Abstract

The application provides a kind of eight-sections palm-chi movement evaluation method and system based on dynamic programming and deep learning, including collecting eight-sections palm-chi movement video data;Through the eight-sections palm-chi movement recognition model based on MediaPipe and DTW, the matching path, DTW distance and single frame score are obtained, the feature sequence of pivot action output by the eight-sections palm-chi movement recognition model is uniformly frame extracted and processed and then input into the eight-sections palm-chi movement evaluation model based on LSTM to obtain the classification label of user pivot action, and the error of user in key action is reflected according to the classification label. The skeleton points are extracted through the Mediapipe framework, the total score is obtained according to the DTW distance, the similarity between the overall movement of user and the standard action is evaluated according to the rhythm score, and the rhythm stability of the movement can be reflected in detail according to the single frame score;The problem of unstable length of video sequence is solved by using the uniform frame extraction method, and the model effectively learns and recognizes the action features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of action recognition, and in particular to a Ba Duan Jin action evaluation method and system based on dynamic programming and deep learning. BACKGROUND

[0002] As an ancient Chinese traditional exercise, Ba Duan Jin has unique health care value and simple and easy characteristics. However, although Ba Duan Jin has many benefits, it is difficult to provide real-time and accurate posture evaluation and guidance to the exercisers due to the lack of effective technical means.

[0003] The traditional teaching method mainly relies on the experience and personal observation of the coach, which is not only inefficient, but also prone to inconsistent guidance due to subjective factors. In order to solve the problems existing in the traditional technology, people try to use smart phones and wearable devices to assist the practice of Ba Duan Jin, but these devices still need to be improved in terms of accuracy and real-time performance. With the development of computer vision and machine learning technology, it is now possible to assist sports training through motion capture and data analysis to improve the standardization of actions and learning efficiency. However, the existing technology has not yet provided a complete recognition and evaluation method for complex sequence actions such as Ba Duan Jin. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a Ba Duan Jin action evaluation method and system based on dynamic programming and deep learning. The present application obtains the matching path, DTW distance and single frame score through the Ba Duan Jin action recognition model, and obtains the feature sequence of the pivotal action according to the matching path. After uniform frame extraction processing is performed on the feature sequence of the pivotal action output by the Ba Duan Jin action recognition model, the classification label of the user pivotal action is obtained by inputting it into the Ba Duan Jin action evaluation model, and the errors of the user in the key actions are reflected according to the classification label.

[0005] The technical scheme of the present application is as follows: a Ba Duan Jin action evaluation method based on dynamic programming and deep learning, comprising the following steps:

[0006] S1), collect Ba Duan Jin action video data, and divide it into a training set and a validation set after labeling;

[0007] S2), construct and train a Ba Duan Jin action recognition model based on MediaPipe and dynamic time warping algorithm DTW, obtain the matching path, DTW distance and single frame score through the trained Ba Duan Jin action recognition model, and obtain the feature sequence of the pivotal action according to the matching path;

[0008] S3), an eight-section exercise action evaluation model based on a long short-term memory network (LSTM) is constructed and trained, a feature sequence of a pivotal action output by the eight-section exercise action recognition model is uniformly frame-sampled and processed, and then input into the trained eight-section exercise action evaluation model to obtain a classification label of the pivotal action of the user, and errors of the user in the key action are reflected according to the classification label;

[0009] S4), the trained eight-section exercise action recognition model and the eight-section exercise action evaluation model are deployed to a cloud server to recognize and evaluate the eight-section exercise action of the user.

[0010] Preferably, in step S1), the collected eight-section exercise action video data of the user is segmented into continuous image segments according to a frame rate, and each image segment represents a posture state of the user at a specific time point.

[0011] Preferably, in step S2), three-dimensional space coordinate data of skeletal points of a human body are recognized from the eight-section exercise action video data by a Pose method based on MediaPipe, and a human body action model is constructed according to the skeletal points of the human body.

[0012] Preferably, in step S2), an angle index of each joint of the human body and a time sequence of the angle index are obtained according to the human body action model.

[0013] Preferably, in step S2), a matching path between a time sequence of the action of the user and a time sequence of a standard action video, a DTW distance and a single-frame score are calculated by a dynamic time warping algorithm (DTW), the matching path is used for recognition of the action of the user and returns a feature sequence of a pivotal action, the DTW distance is output as a total similarity score of the movement by a scoring function, and a score result of the rhythm of the action of the user is obtained by the single-frame score.

[0014] Preferably, in step S2), the matching path is used for action recognition and returns a feature sequence of a pivotal action.

[0015] The calculation method of the single-frame score is that a cosine distance between each frame of the movement action of the user and a standard frame matched therewith is calculated, a scoring function is set according to the property that the maximum value of the cosine distance is 2 to obtain a single-frame score score of each frame of the action of the user. d That is,

[0016]

[0017] In the formula, sim(A, B) represents a cosine similarity between an angle index A of a standard action and an angle index B of a user action.

[0018] As preferred, in step S2), the DTW distance is converted into the overall similarity score score of the movement by using a scoring function dwt , the overall similarity score score dwt The calculation formula is as follows:

[0019]

[0020] Wherein, d represents the DTW distance between the current user action time sequence and the standard action time sequence; d max represents the cosine distance when the similarity is the lowest; d great is the DTW distance approximation according to the excellent test action, when d < d great , the action is very excellent.

[0021] As preferred, in step S2), the rhythm score result score of the user action is calculated by single frame scoring r , which is represented as:

[0022]

[0023] In the formula, S represents the frequency of the user single frame score score d greater than 60 points; D is the total frame number of the user's pivot action.

[0024] As preferred, in step S3), the uniform frame extraction processing is: for a given target sequence length and original feature sequence length, the frame extraction interval is calculated, and then the original feature sequence is uniformly extracted according to the frame extraction interval.

[0025] As preferred, in step S3), for the original feature sequence length seq_len, the given target sequence length target_len; then the frame extraction interval interval = seq_len / target_len.

[0026] As preferred, in step S3), for the frame extraction interval interval, starting from the 0th frame of the original feature sequence, every interval frame is extracted until the target sequence length target_len is reached, and finally the remaining frames are processed. If the original feature sequence length seq_len cannot be divided by the target sequence length target_len, the last frame of the original feature sequence is ignored.

[0027] As preferred, in step S3), the single frame features after uniform frame extraction are input into the Baduanjin action evaluation model based on long short-term memory network LSTM in time sequence, the hidden state is learned and updated through internal control mechanism, so as to capture the time dependence of Baduanjin action, and the classification of user action is output.

[0028] As preferred, the application also provides an eight-section exercise action evaluation system based on dynamic programming and deep learning, comprising:

[0029] a data acquisition module for acquiring eight-section exercise action video data of a user;

[0030] a human body skeleton point recognition module for recognizing three-dimensional space coordinate data of skeleton points of a human body from the eight-section exercise action video data by using a Pose method based on MediaPipe, and constructing a human body action model according to the human body skeleton points;

[0031] a dynamic time warping module for calculating a matching path, a DTW distance and a single-frame score between a time sequence of a user action and a time sequence of a standard action video, obtaining a feature sequence of a user pivotal action according to the matching path, obtaining a total similarity score of a user action according to the DTW distance, and obtaining a score result of a user action rhythm through the single-frame score;

[0032] an action evaluation module for capturing a time-dependent relationship of an eight-section exercise action from single-frame features processed by uniform frame extraction according to an eight-section exercise action evaluation model based on a long short-term memory network (LSTM), and obtaining a classification result of a user action.

[0033] As preferred, the calculation method of the single-frame score is:

[0034] a cosine distance calculation is performed on each frame of a user action and a standard frame matched therewith, a score function is set according to the property that the maximum value of a cosine distance is 2 to obtain a single-frame score score of each frame of the user action d , that is,

[0035]

[0036] In the formula, sim(A, B) represents a cosine similarity between an angle index A of a standard action and an angle index B of a user action.

[0037] As preferred, the DTW distance is converted into a total similarity score score of the action by using a score function dwt , and the total similarity score score dwt is calculated according to the following formula:

[0038]

[0039] wherein d represents a DTW distance between a current user action time sequence and a standard action time sequence; d max represents a cosine distance when the similarity is the lowest; d great is an approximation according to the DTW distance of an excellent test action, and when d < d great , the action is very excellent.

[0040] As preferred, the rhythm score result score of the user action is calculated by single frame score r , expressed as:

[0041]

[0042] In the formula, S represents the single frame score score of the user d The frequency of more than 60 points; D the total number of frames of the user's pivotal action.

[0043] As preferred, the uniform frame extraction processing is: for a given target sequence length and original feature sequence length, the frame extraction interval is calculated, and then the original feature sequence is uniformly extracted according to the frame extraction interval.

[0044] The beneficial effects of the present application are:

[0045] 1、The present application extracts the skeleton points through the Mediapipe framework, takes the joint angle index as the time sequence element, calculates the time sequence normalization with the sequence of the standard video, takes the matching path, and inversely assigns the user video action label according to the index of the standard action video time sequence, so as to realize the recognition effect;

[0046] 2、The present application obtains the total score according to the DTW distance, evaluates the similarity of the user's overall movement and the standard action according to the rhythm score, calculates the cosine distance between each frame of the user's movement and the matching standard frame, sets the score function according to the property that the maximum value of the cosine distance is 2 to obtain the single frame score of each frame, and carries out statistics according to the score interval; so as to obtain the single frame score score segment distribution, and further reflect the rhythm stability of the action in detail, and further use the entropy as a normalized feature to evaluate the rhythm of the user's action, so as to objectively quantify the consistency and stability of the action;

[0047] 3、The present application solves the problem of unstable length of the video sequence by using the uniform frame extraction method, and ensures that the model can effectively learn and recognize the action features. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 It is the flow framework diagram of the method of the present application;

[0049] Figure 2 It is the schematic diagram of 33 skeleton points of the MediaPipe recognition of the present application;

[0050] Figure 3 It is the schematic diagram of the human action model constructed by the present application;

[0051] Figure 4 It is the flow schematic diagram of the uniform frame extraction of the present application;

[0052] Figure 5 Structure diagram output for the present application;

[0053] Figure 6 Structure framework diagram of the system of the present application. DETAILED DESCRIPTION

[0054] The specific embodiments of the present application will be further described below in conjunction with the accompanying drawings:

[0055] Example 1

[0056] As Figure 1 shown, the embodiment provides a Ba Duan Jin movement evaluation method based on dynamic programming and deep learning, including the following steps:

[0057] S1), collect Ba Duan Jin movement video data, and divide it into training set and validation set after labeling;

[0058] In this embodiment, Ba Duan Jin movement video data under different crowds and different environments is collected, wherein the labeled content includes action category, start time, end time, and action standard degree information; and the collected Ba Duan Jin movement video data is segmented into continuous image segments according to the frame rate, and each image represents the posture state of the user at a certain time point.

[0059] In this embodiment, the eight moves of Ba Duan Jin are: two hands supporting the sky to regulate Sanjiao, left and right opening the bow to shoot the hawk, regulating the spleen and stomach to single lift, looking back at the five labors and seven injuries, shaking the head and tail to remove heart fire, two hands climbing the foot to fix the kidney waist, clenched fist and angry eyes to increase strength, and seven back flips to eliminate diseases, as shown in Table 1:

[0060] Table 1 Eight moves of Ba Duan Jin

[0061]

[0062]

[0063] The frame number of the extracted standard video is shown in Table 2.

[0064] Table 2 Frame number of standard video

[0065]

[0066] S2), build and train a Ba Duan Jin movement recognition model based on MediaPipe and dynamic time warping algorithm DTW, obtain matching path, DTW distance and single frame score through the trained Ba Duan Jin movement recognition model, and obtain the feature sequence of the pivotal action according to the matching path.

[0067] As Figure 2 and 3As shown, the embodiment analyzes each frame of image based on the MediaPipe-based Pose method, extracts 33 skeleton point information of the user's arms, elbows, knees, ankles and shoulders from the image, and forms three-dimensional coordinate data; then calculates the included angle between the skeletons connected between the hand and leg skeleton points of each image segment. And construct a human action model according to 33 human skeleton points. According to the human action model, the angle index of each joint of the human body and the time sequence of the angle index are obtained.

[0068] Then the dynamic time warping algorithm DTW is used to calculate the matching path, DTW distance and single frame score between the time sequence of the user's action and the time sequence of the standard action video, the matching path is used for user action recognition, and the feature sequence of the pivotal action is returned, the DTW distance is output as the overall similarity score of the move by the scoring function, and the scoring result of the user's action rhythm is obtained by the single frame score.

[0069] The embodiment takes "left and right open bow like shoot hawk" as an example, and identifies three actions of "wrist touch", "open bow" and "parallel step" through the matching path, wherein the "open bow" action is the pivotal action, and the error points are defined.

[0070] The matching path is used for action recognition, and the feature sequence of the pivotal action is returned.

[0071] The DTW distance calculation method is:

[0072]

[0073] In the formula, d represents the DTW distance between the current user action time sequence and the standard action time sequence; A represents the angle index set of the standard action; B represents the angle index set of the user action; a i is the i-th joint angle in the standard action; b i is the i-th joint angle in the user action; and Φ represents the frame number set of the standard video and the user action video.

[0074] The calculation method of the single frame score is: the cosine distance between each frame of the user's move action and the matching standard frame is calculated, and the scoring function is set according to the property that the maximum value of the cosine distance is 2 to obtain the single frame score score d , that is:

[0075]

[0076] In the formula, sim(A, B) represents the cosine similarity between the angle index A of the standard action and the angle index B of the user action.

[0077] Wherein, the expression of cosine similarity sim(A, B) is:

[0078] sim(a i ,b j )=1-cos(a i ,b j )

[0079] A={a1,a2,a3,…,a n}

[0080] B={b1,b2,b3,…,b m}

[0081] In the formula, a i is the i th joint angle in the standard action; b i represents the i th joint angle in the user action, n represents the frame number of the standard action video, and m is the frame number of the user action.

[0082] The DTW distance is converted into the overall similarity score score dwt of the move by using the scoring function, and the overall similarity score score dwt is calculated as follows:

[0083]

[0084] Wherein, d represents the DTW distance between the current user action time sequence and the standard action time sequence; d max represents the cosine distance when the similarity is the lowest; d great is the DTW distance approximation according to the excellent test action, and when d < d great , the action is very excellent. As shown in Table 3, the total score results of the four output videos are shown.

[0085] The rhythm score score r of the user action is calculated by single frame scoring, which is represented as:

[0086]

[0087] In the formula, S represents the frequency of the user single frame score score d greater than 60 points; and D represents the total frame number of the user pivotal action.

[0088] Table 3: Total score results of four videos

[0089]

[0090]

[0091] S3), an eight-section exercise action evaluation model based on a long short-term memory network LSTM is constructed and trained, a feature sequence of a pivotal action output by the eight-section exercise action recognition model is uniformly frame-sampled, and then is input into the trained eight-section exercise action evaluation model to obtain a classification label of a pivotal action of the user, and errors of the user on the key action are reflected according to the classification label;

[0092] In this embodiment, the video category is encoded by One-hot encoding. The encoding is not only the label of the training data, but also a binary encoding. In the four error points of the "open bow action" in the "left and right open bow like shooting flying eagle", the encoding is a 4-bit binary number. 1 indicates that the error point has an error, and 0 indicates that the error point has no error. In the data labeling, the labeler arranges them in order from left to right to form a binary number and convert it into a decimal number as a label. Correspondingly, the output classification result of the eight-section exercise action evaluation model based on the long short-term memory network LSTM is a tensor of (1, 4), wherein each element represents the confidence of classification as 1, and the value is between 0 and 1. When calculating the loss function, the label is converted into one-hot encoding, and the cross-entropy is calculated with the output tensor to realize back propagation and training.

[0093] In this embodiment, the uniform frame sampling processing is: for a given target sequence length and an original feature sequence length, the frame sampling interval is calculated, and then the original feature sequence is uniformly frame-sampled according to the frame sampling interval. Specifically,

[0094] As shown in Figure 4 For the original feature sequence length seq_len and the given target sequence length target_len, the frame sampling interval interval = seq_len / target_len. For the frame sampling interval interval, a frame is extracted every interval frames from the 0th frame of the original feature sequence, until the target sequence length target_len is reached. Finally, the remaining frames are processed. If the original feature sequence length seq_len cannot be divided by the target sequence length target_len, the last frame of the original feature sequence is ignored. For example, the "open bow" action is reduced to 10 frames by uniform frame sampling.

[0095] Then, the single-frame features processed by uniform frame sampling are input into the eight-section exercise action evaluation model based on the long short-term memory network LSTM in time sequence to learn and update the hidden state through the internal control mechanism, so as to capture the time dependence of the eight-section exercise action, and output the classification of the user action.

[0096] In this embodiment, the parameters of the eight-section exercise action evaluation model based on the long short-term memory network LSTM are shown in Table 4.

[0097] Table 4. Parameters of the eight-section exercise action evaluation model based on the long short-term memory network (LSTM)

[0098]

[0099] During the training process, the eight-section exercise action evaluation model outputs a tensor with a shape of (1, 16) normalized by the Softmax function. The cross-entropy calculation is performed with the one-hot encoded label encoding as the loss function. Through the backpropagation algorithm, the weights and bias parameters of the LSTM network are continuously adjusted to minimize the loss function, so that the model can better fit the training data. The Adam optimizer is combined to accelerate the model training process and improve the convergence speed. Finally, the output result of the eight-section exercise action evaluation model is the classification code of the video, and the corresponding text content is fed back to the exerciser. In this embodiment, the precision, sensitivity, recall rate, and F1 value are used to evaluate the performance of the model. The performance index test results of the eight-section exercise action evaluation model are shown in Table 5.

[0100] Table 5. Performance index test results of the eight-section exercise action evaluation model

[0101]

[0102]

[0103] S4), the trained eight-section exercise action recognition model and eight-section exercise action evaluation model are deployed to a cloud server to recognize and evaluate the user's eight-section exercise actions. The final output result of this embodiment is shown in Table 6. Figure 5

[0104] Embodiment 2

[0105] As shown in Table 7, this embodiment provides an eight-section exercise action evaluation system based on dynamic programming and deep learning, which includes: Figure 6

[0106] A data acquisition module for acquiring eight-section exercise action video data of a user;

[0107] A human body skeleton point recognition module for recognizing three-dimensional space coordinate data of 33 human body skeleton points from the eight-section exercise action video data using the Pose method based on MediaPipe, and constructing a human body action model according to the 33 human body skeleton points;

[0108] ​​a dynamic time warping module, configured to calculate a matching path, a DTW distance and a single-frame score between a time sequence of a user action and a time sequence of a standard action video, and obtain a feature sequence of a user pivot action according to the matching path, and obtain a total similarity score of a user movement according to the DTW distance, and obtain a score result of a user action rhythm through the single-frame score;

[0109] an action evaluation module, configured to capture a time-dependent relationship of the Qigong action according to an eight-section Qigong action evaluation model based on a long short-term memory network (LSTM) from the single-frame features processed by the uniform frame extraction, and obtain a classification result of the user action.

[0110] Preferably, the single-frame score is calculated in the following manner:

[0111] a cosine distance between each frame of the user movement and a standard frame matched with the frame is calculated, and a single-frame score score of each frame of the user movement is obtained according to a score function based on a property that the cosine distance is 2. d That is,

[0112]

[0113] In the formula, sim(A, B) represents a cosine similarity between an angle index A of a standard action and an angle index B of a user action.

[0114] Preferably, the DTW distance is converted into a total similarity score score of the movement by using the score function dwt , and the total similarity score score dwt is calculated according to the following formula:

[0115]

[0116] In the formula, d represents a DTW distance between a current user action time sequence and a standard action time sequence; d max represents a cosine distance when the similarity is the lowest; d great is an approximate DTW distance of an excellent test action, and when d < d great , the action is very excellent.

[0117] Preferably, a rhythm score result score r of the user action is calculated through the single-frame score, and is represented as:

[0118]

[0119] In the formula, S represents a frequency of the user single-frame score score d being greater than 60 points; and D represents a total frame number of the pivot action of the user.

[0120] As preferred in the embodiment, the uniform frame extraction processing is that, for a given target sequence length and original feature sequence length, the frame extraction interval is calculated, and then the original feature sequence is uniformly extracted according to the frame extraction interval.

[0121] The above embodiments and descriptions are only to illustrate the principles and the best embodiments of the present application, and the present application can have various changes and improvements without departing from the spirit and the scope of the present application, and all the changes and improvements fall within the scope of the present application.

Claims

1. A method for evaluating Baduanjin movements based on dynamic programming and deep learning, characterized in that, Comprise the following steps: S1), collect Ba Duan Jin action video data, and divide it into training set and validation set after labeling; S2), build and train Ba Duan Jin action recognition model based on MediaPipe and dynamic time warping algorithm DTW, get matching path, DTW distance and single frame score through the trained Ba Duan Jin action recognition model, and get the feature sequence of pivotal action according to the matching path; Specifically as follows: The dynamic time warping algorithm DTW is used to calculate the matching path, the DTW distance and the single frame score between the time sequence of the user's action and the time sequence of the standard action video, the matching path is used for the recognition of the user's action, and the feature sequence of the pivotal action is returned, the DTW distance is output as the overall similarity score of the movement through the scoring function, and the scoring result of the user's action rhythm is obtained through the single frame score; The cosine distance is calculated for each frame of the user's gesture and the standard frame matched therewith, and a scoring function is set according to the property that the maximum value of the cosine distance is 2 to obtain a single-frame score of each frame of the user's gesture That is: ； In the formula, represents the cosine similarity between the angle index A of the standard action and the angle index B of the user action. The DTW distance is converted to an overall similarity score for the move using a scoring function , overall similarity score The formula is as follows: ； wherein, DTW distance between the current user action time series and the standard action time series; cosine distance when the similarity is the lowest; is the DTW distance approximation according to the excellent test action, when the action is very excellent; Calculating a rhythm score result of a user action by single-frame scoring is represented as: ； In the formula, representing the user single-frame score the frequency of scores greater than 60; the total number of hub actions of the user S3), build and train Ba Duan Jin action evaluation model based on long short-term memory network LSTM, uniformly frame process the feature sequence of pivotal action output by Ba Duan Jin action recognition model, then input into the trained Ba Duan Jin action evaluation model to get the classification label of user pivotal action, and reflect the user's error in key action according to the classification label; S4), deploy the trained Ba Duan Jin action recognition model and Ba Duan Jin action evaluation model to the cloud server to recognize and evaluate the user's Ba Duan Jin action.

2. The method according to claim 1, wherein the method is characterized in that: In step S2, the three-dimensional space coordinate data of the human body's skeletal points is recognized from the Ba Duan Jin action video data based on the Pose method of MediaPipe, and a human body action model is constructed according to the human body skeletal points.

3. The method according to claim 2, wherein the method is characterized in that: In step S2, the angle indexes of each joint of the human body and the time sequence of the angle indexes are obtained according to the human body action model.

4. The eight-section exercise movement evaluation method based on dynamic programming and deep learning according to claim 1, characterized in that: In step S3, the uniform frame processing is: for a given target sequence length and original feature sequence length, the frame interval is calculated, and then the original feature sequence is uniformly framed according to the frame interval; The single frame features after uniform frame processing are input into the Ba Duan Jin action evaluation model based on long short-term memory network LSTM in time sequence to learn and update the hidden state through internal control mechanism, so as to capture the time dependence of Ba Duan Jin action, and output the classification of user action.

5. The Baduanjin movement evaluation system based on dynamic programming and deep learning, characterized in that, The system uses the method of any one of claims 1-4 to evaluate the user's Ba Duan Jin action, and the system comprises: A data acquisition module for acquiring Ba Duan Jin action video data of a user; A human body skeletal point recognition module for recognizing three-dimensional space coordinate data of human body skeletal points from Ba Duan Jin action video data based on the Pose method of MediaPipe, and constructing a human body action model according to the human body skeletal points; A dynamic time warping module for calculating the matching path, the DTW distance and the single frame score between the time sequence of the user's action and the time sequence of the standard action video, and obtaining the feature sequence of the user's pivotal action according to the matching path, and obtaining the overall similarity score of the user's movement through the DTW distance, and obtaining the scoring result of the user's action rhythm through the single frame score; The action evaluation module is configured to capture time-dependent relationships of the eight-section exercise actions from the single-frame features processed by uniform frame extraction according to an eight-section exercise action evaluation model based on a long short-term memory network (LSTM), and obtain a classification result of the user actions.

6. The eight-section exercise action evaluation system according to claim 5, characterized in that: The calculation method of the single-frame score is: The cosine distance is calculated for each frame of the user's gesture and the standard frame matched therewith, and a scoring function is set according to the property that the maximum value of the cosine distance is 2 to obtain a single-frame score of each frame of the user's gesture That is, In the formula, denotes the cosine similarity between the angle index A of the standard action and the angle index B of the user action; The DTW distance is converted to an overall similarity score for the move using a scoring function , overall similarity score The formula is as follows: wherein, represents the DTW distance between the current user action time series and the standard action time series; represents the cosine distance when the similarity is the lowest; is approximated from the DTW distance of the excellent test action, when the action is very excellent; Calculating a rhythm score result of a user action by single-frame scoring is represented as: In the formula, representing the user single-frame score Frequency of scores greater than 60 points; Total number of hub actions of the user

Images