Human body motion evaluation method, evaluation device and evaluation system

Abstract

The present application provides a human motion evaluation method, evaluation device and evaluation system, and the evaluation method includes the following steps: training and deploying a human motion evaluation model; using the human motion evaluation model to evaluate the user's human motion and obtain a feature score; using the sentiment analysis model to analyze the user's sentiment when viewing the feature score output by the human motion evaluation model, and constructing an error matching library based on the analysis results; using the error matching library to update the human motion evaluation model; using the updated human motion evaluation model to evaluate the human motion. The present application adopts the measurement method of deep learning and sentiment analysis, combined with the traditional measurement method, to obtain a motion evaluation method that can perform feature measurement learning by itself, find effective features by itself, and automatically learn and evolve when an unseen motion occurs.

Description

Technical Field

[0001] The present application belongs to the technical field of human motion analysis, and specifically relates to a human motion evaluation method, evaluation device and evaluation system. Background Art

[0002] With the advancement of deep learning technology and high-performance technology, real-time human posture assessment and behavior understanding have become research hotspots in the current field of computer vision. Some research results in the field of computer vision have been successfully applied to many fields, for example, detecting elderly falls through videos. In addition, human motion assessment and similarity evaluation also have broad application prospects in tasks such as sports video analysis and physical education. Human behavior is composed of a set of time-series actions. Effectively judging the gap between standard actions (coach actions) and input actions (student actions) is an important task to measure the similarity between standard actions and input actions.

[0003] At present, human action similarity assessment mainly adopts the following two methods:

[0004] (1) Use an RGB camera or RGB-D camera to extract human skeleton data, design some simple feature indicators based on some action essentials, and use the feature indicators to evaluate human actions. For example, in table tennis hitting, elbow reverse action is not allowed. This method is used to calculate the coordinates of the elbow joint and the wrist joint. If the coordinates are greater than the preset threshold, it is determined to be an artificial elbow reverse action.

[0005] This method summarizes complex human behavior into several instantaneous states, thereby quantifying the differences and obtaining the similarity of the actions. However, it ignores the fact that human action is a continuous process, and the sampling of several instantaneous states is far from being summarized as a continuous human action. At the same time, when the wrong action is not in the set summary action essentials, the system will not be able to distinguish it.

[0006] (2) Use RGB cameras or RGB-D cameras to extract human skeleton data, extract some feature indicators that can reflect human motion posture, and then use pure mathematical description method - metric learning to evaluate the difference of feature indicators. For example, design joint plane features, and then use DTW method or Euclidean distance similarity method to compare standard motion and input motion to obtain the difference.

[0007] This method extracts some action characteristics at each moment and quantifies the action characteristics into features, and then uses the metric learning method to compare the features at each moment to obtain the similarity of the actions. An action is a change pattern that affects the moment t+1. Using simple metric learning comparison cannot well discover this influence pattern. At the same time, in the process of quantifying features, it is very likely that some designed features cannot reflect some special actions. When encountering an action that cannot be distinguished by features, the system cannot distinguish it, which will cause a large evaluation error.

[0008] Both of the above methods hope to find feature measurements that can accurately summarize complex human motion behaviors, and then use pure mathematical methods (that is, measurement methods) to compare these feature measurements and give results. However, some sports movements are not just the result of some instantaneous behaviors, but the result of a combination of movements over a period of time. It is difficult for artificially designed motion feature measurements to fully summarize all human motion behaviors. In addition, using these methods, once an unrecognizable situation occurs, it is impossible to automatically correct the recognition. It is necessary to manually analyze the feature measurements of the unrecognized movements, and then add the feature measurements to the system before the system can recognize such movements. Summary of the invention

[0009] In order to overcome the problems existing in the related art to at least a certain extent, the present application provides a human motion evaluation method, an evaluation device and an evaluation system.

[0010] According to a first aspect of an embodiment of the present application, the present application provides a method for evaluating human motion, which comprises the following steps:

[0011] Train and deploy a human motion evaluation model;

[0012] The human motion evaluation model is used to evaluate the user's human motion and obtain a feature score;

[0013] The sentiment analysis model is used to analyze the sentiment of users when viewing the feature scores output by the human action evaluation model, and a wrong matching library is constructed based on the analysis results;

[0014] The human motion evaluation model is updated using the error matching library;

[0015] The updated human motion evaluation model is used to evaluate human motion.

[0016] In the above human motion evaluation method, the specific process of training and deploying the human motion evaluation model is:

[0017] The process of generating an action evaluation dataset based on human skeleton data is as follows:

[0018] Use RGB-D camera to obtain human body two-dimensional video stream and depth image stream;

[0019] Generate 3D human skeleton data based on human 2D video stream and depth image stream;

[0020] Obtain input action sequence according to 3D human skeleton data;

[0021] Perform feature quantification and measurement comparison on the input action sequence and the standard action sequence to obtain feature scores;

[0022] According to the standard action sequence and the input action sequence and the feature scores corresponding to the standard action sequence and the input action sequence, a human action evaluation sample is obtained;

[0023] Screening each human action evaluation sample to obtain an action evaluation data set;

[0024] The deep learning model is trained using human motion evaluation samples in the motion evaluation dataset to obtain and deploy the human motion evaluation model.

[0025] The specific process of using the human motion evaluation model to evaluate the user's human motion and obtain the feature score in the above human motion evaluation method is as follows:

[0026] Obtaining the user's real-time input action sequence and the standard action sequence selected by the user to be learned;

[0027] Inputting the acquired real-time input action sequence and standard action sequence into the human action evaluation model;

[0028] The human motion evaluation model evaluates the similarity between the input motion sequence and the standard motion sequence and outputs a feature score.

[0029] In the above-mentioned human action evaluation method, the process of using the sentiment analysis model to analyze the sentiment of the user when viewing the feature score output by the human action evaluation model, and constructing the error matching library according to the analysis results is as follows:

[0030] Use an RGB-D camera to collect the video stream of the user's expression when seeing the feature score output by the human action evaluation model in real time, and send the collected video stream to the sentiment analysis model;

[0031] The sentiment analysis model analyzes the emotions of the user when seeing the feature scores output by the human action evaluation model, and obtains the probability of the user's emotions when seeing the feature scores; the emotions include happiness, confusion, and frustration;

[0032] When doubt has the highest probability among the three emotions, the sentiment analysis model compares the obtained doubt probability with the preset doubt probability threshold, and calculates the adjusted score based on the difference between the two and the adjustment base;

[0033] The input action sequence and standard action sequence corresponding to when the user's emotion is doubtful when seeing the feature score, as well as the adjusted score, are used as error samples, and the error matching library is constructed using the error samples.

[0034] Furthermore, the specific process of updating the human motion evaluation model using the error matching library is as follows:

[0035] Input the samples in the wrong matching library into the human action evaluation model;

[0036] The weights of the human motion evaluation model are updated through the back propagation algorithm so that the loss function of the human motion evaluation model is reduced. When the loss function of the human motion evaluation model converges, the updating is stopped and the obtained human motion evaluation model is used as the updated human motion evaluation model.

[0037] According to a second aspect of the embodiments of the present application, the present application provides a human motion evaluation device, which includes a human motion evaluation model, a sentiment analysis model and an error matching library;

[0038] The human motion evaluation model is used to evaluate the similarity between the acquired real-time input motion sequence of the user and the standard motion sequence selected by the user to be learned, and output a feature score;

[0039] The sentiment analysis model is used to analyze the sentiment of the user when viewing the feature scores output by the human action evaluation model, and to build an error matching library based on the analysis results;

[0040] The error matching library is used to update the human motion evaluation model.

[0041] Furthermore, the human motion evaluation model is obtained by prior training and deployment; the specific process of obtaining the human motion evaluation model is:

[0042] Generating an action evaluation dataset based on human skeleton data includes the following steps:

[0043] Use RGB-D camera to obtain human body two-dimensional video stream and depth image stream;

[0044] Generate 3D human skeleton data based on human 2D video stream and depth image stream;

[0045] Obtain input action sequence according to 3D human skeleton data;

[0046] Obtain standard action sequences from a standard action database;

[0047] Perform feature quantification and measurement comparison on the input action sequence and the standard action sequence to obtain feature scores;

[0048] According to the standard action sequence and the input action sequence and the feature scores corresponding to the standard action sequence and the input action sequence, a human action evaluation sample is obtained;

[0049] Screening each human action evaluation sample to obtain an action evaluation data set;

[0050] The deep learning model is trained using the human motion evaluation samples in the motion evaluation dataset to obtain a human motion evaluation model.

[0051] Furthermore, the sentiment analysis model analyzes the sentiment of the user when viewing the feature scores output by the human action evaluation model, and the specific process of constructing the error matching library according to the analysis results is as follows:

[0052] Use an RGB-D camera to collect the video stream of the user's expression when the user sees the feature score output by the human action evaluation model corresponding to the user's real-time input action sequence, and send the collected video stream to the sentiment analysis model;

[0053] The sentiment analysis model analyzes the user's sentiment when seeing the feature score output by the human action evaluation model, and obtains the probability of the user's sentiment when seeing the feature score, wherein the sentiment includes happiness, confusion and frustration;

[0054] When doubt has the highest probability among the three emotions, the sentiment analysis model compares the obtained doubt probability with the preset doubt probability threshold, and calculates the adjusted score based on the difference between the two and the adjustment base;

[0055] The input action sequence and standard action sequence corresponding to when the user's emotion is doubtful when seeing the feature score, as well as the adjusted score, are used as error samples, and the error matching library is constructed using the error samples.

[0056] Furthermore, the specific process of the error matching library updating the human motion evaluation model is as follows:

[0057] Input the samples in the wrong matching library into the human action evaluation model;

[0058] The weights of the human motion evaluation model are updated through the back propagation algorithm so that the loss function of the human motion evaluation model is reduced. When the loss function of the human motion evaluation model converges, the updating is stopped and the obtained human motion evaluation model is used as the updated human motion evaluation model.

[0059] According to a third aspect of the embodiments of the present application, the present application provides a human motion evaluation system, which includes an RGB-D camera, a human-computer interaction interface, and a human motion evaluation device;

[0060] The RGB-D camera is used to collect a two-dimensional video stream and a depth image stream of a human body, and is also used to collect a video stream of an expression of a user when he sees a feature score output by a human action evaluation model in a human action evaluation device;

[0061] The user selects a standard action sequence to be learned through the human-computer interaction interface, and the human-computer interaction interface presents the feature scores output by the human action evaluation model to the user;

[0062] The human motion evaluation device is used to construct an error matching library according to the analysis results of the sentiment analysis model, and use the error matching library to update the human motion evaluation model, and use the updated human motion evaluation model to evaluate the user's real-time input motion.

[0063] According to the above specific implementation of the present application, it can be seen that there are at least the following beneficial effects:

[0064] Traditional human action annotation usually uses a method where people observe the action and then give a score. This method is too subjective and loses the physical objectivity in action similarity. Compared with the traditional data annotation method, in this application, an objective knowledge system evaluation combined with manual screening is used to obtain an action evaluation dataset.

[0065] Traditional human motion evaluation methods use two independent steps: human motion feature quantification and metric comparison. Both steps require human experts to design. The designed methods cannot meet the needs of all actions, and there is always a large part of the actions that cannot be well recognized. Compared with this traditional human motion evaluation method, the human motion evaluation model based on the deep learning model in this application couples feature quantification and metric comparison well, and has a higher recognition accuracy.

[0066] The traditional human motion evaluation method is an open-loop evaluation method, which cannot supervise whether the score given by the model is appropriate. Compared with this traditional evaluation method, this application makes the human motion evaluation a closed-loop evaluation method by adding a sentiment analysis model, so that the scoring satisfies both the physical objective properties and the ability of human senses.

[0067] After the traditional human motion evaluation method is deployed on the device, the evaluation model is a solidified model that cannot self-repair errors. The model can only be manually upgraded when a command is manually sent to upgrade the model. Compared with this traditional human motion evaluation method, this application adds a full-life learning and updating process for the human motion evaluation model, which can automatically correct the human motion evaluation model after each error, ensuring that the same error will not be generated next time the same error is encountered, allowing the device to maintain the ability of continuous self-learning and automatic model optimization throughout its entire service life.

[0068] It should be understood that the above general description and the following specific embodiments are only exemplary and illustrative and do not limit the scope of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The attached drawings below are part of the specification of the present application, which illustrate embodiments of the present application. The attached drawings together with the description of the specification are used to explain the principles of the present application.

[0070] Figure 1 This is a main flow chart of a human motion evaluation method provided in an embodiment of the present application.

[0071] Figure 2 A detailed flow chart of a human body motion evaluation method provided in an embodiment of the present application

[0072] Figure 3 A schematic diagram of the process of obtaining a motion evaluation data set in a human motion evaluation method provided in an embodiment of the present application.

[0073] Figure 4 A schematic diagram of the process of training a human motion evaluation model in a human motion evaluation method provided in an embodiment of the present application.

[0074] Figure 5 A schematic diagram of the process of constructing an error matching library in a human motion evaluation method provided in an embodiment of the present application.

[0075] Figure 6 A schematic diagram of the process of updating a human motion evaluation model by using an error matching library in a human motion evaluation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clearly understood, the spirit of the contents disclosed in the present application will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present application, any technician in the relevant technical field can change and modify them according to the techniques taught by the contents of the present application without departing from the spirit and scope of the contents of the present application.

[0077] The exemplary embodiments and descriptions of the present application are used to explain the present application, but are not intended to limit the present application. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0078] The terms “first”, “second”, etc. used in this document do not particularly refer to an order or sequence, nor are they used to limit this application. They are only used to distinguish elements or operations described with the same technical terms.

[0079] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0080] As used herein, "and / or" includes any or all combinations of the items described.

[0081] As used herein, “plurality” includes “two” and “more than two”; as used herein, “plurality groups” includes “two groups” and “more than two groups”.

[0082] Certain terms used to describe the present application are discussed below or elsewhere in this specification to provide additional guidance to those skilled in the art regarding the description of the present application.

[0083] Figure 1 This is a main flow chart of a human motion evaluation method provided in an embodiment of the present application. Figure 2 A detailed flow chart of a human motion evaluation method provided in an embodiment of the present application.

[0084] like Figure 1 and Figure 2 As shown, the human body motion evaluation method provided by the present application comprises the following steps:

[0085] S1. Train and deploy the human motion evaluation model. The specific process is as follows:

[0086] S11, generate action evaluation data set based on human skeleton data, such as Figure 3 As shown, the specific process is:

[0087] S111. Use an RGB-D camera to obtain a two-dimensional video stream and a depth image stream of a human body.

[0088] Specifically, the RGB-D camera includes an image acquisition sensor and a corresponding depth collector. The image acquisition sensor is used to acquire a two-dimensional video stream of the human body; and the depth collector is used to acquire a depth image stream.

[0089] S112, generating 3D human skeleton data according to the two-dimensional human body video stream and the depth image stream.

[0090] Specifically, first, the pixel coordinates in the image are obtained according to the RGB image in the two-dimensional video stream of the human body; then, 3D human skeleton data is generated according to the point cloud mapping relationship and the depth information in the depth image stream.

[0091] S113, obtaining an input action sequence according to the 3D human skeleton data.

[0092] Specifically, the input action sequence can be represented as the input skeleton point coordinates of the student.

[0093] The standard action sequence is obtained from the standard action database. Specifically, the standard action sequence can be represented by the pre-stored skeleton point coordinates of the coach.

[0094] S114, perform feature quantification and metric comparison on the input action sequence and the standard action sequence to obtain feature scores, the specific process of which is as follows:

[0095] The traditional motion feature quantification method is used to quantify human motion into features such as joint vectors, joint planes, joint velocities, joint angles, and joint relative positions. It should be noted that the human motion here includes input motion and standard motion.

[0096] The measurement method is used to measure and compare the features corresponding to the input action and the standard action to obtain a feature score. It should be noted that the measurement method can use DTW distance, Euclidean distance, cosine distance, Mahalanobis distance or Manhattan distance to calculate the difference between the quantized features of the input action and the corresponding features of the quantized standard action.

[0097] S115. Use the method of step S114 above to evaluate a large number of human body movements to obtain a large number of human body movement evaluation samples, each of which is in the form of [standard movement sequence, input movement sequence, feature score].

[0098] S116. Screen the human motion evaluation samples to obtain a motion evaluation data set.

[0099] It should be noted that when each human motion evaluation sample is manually screened, an error threshold is preset, and a professional motion scoring coach evaluates the similarity between the input motion and the standard motion according to the motion similarity guidance standard and gives a manual score; and compares the manual score with the feature score. If the error between the two is greater than the preset error threshold, the human motion evaluation sample corresponding to the feature score is discarded, and finally the human motion evaluation sample that meets the above judgment rules is obtained. All human motion evaluation samples that meet the above judgment rules constitute the motion evaluation data set.

[0100] In the process of generating a motion evaluation dataset based on human skeletal data, traditional human motion feature quantification and measurement comparison are used as the physical objective knowledge system, and then human sensory knowledge is added through manual screening. Through physical objective evaluation and manual screening of sensory evaluation, a high-quality motion evaluation dataset can be constructed. This motion evaluation dataset takes into account both objective evaluation and human sensory comfort.

[0101] S12. Use the human action evaluation samples in the action evaluation data set to train the deep learning model to obtain a human action evaluation model, such as Figure 4 As shown, the specific process is:

[0102] A deep learning model based on convolutional neural network and temporal neural network is used to learn the human motion evaluation samples in the motion evaluation dataset. The input of the deep learning model is standard motion sequence, input motion sequence and feature score.

[0103] The use of deep learning models can learn the advantages and disadvantages of traditional human motion evaluation, and at the same time complete the end-to-end structure from input motion sequence to output feature score, without the need to specially design feature quantization and metric comparison models.

[0104] S2. Use the human motion evaluation model to evaluate the user's human motion and obtain a feature score. The specific process is as follows:

[0105] S21, obtaining the user's real-time input action sequence and the standard action sequence selected by the user to be learned;

[0106] Specifically, the user's human body two-dimensional video stream and depth image stream can be obtained in real time through an RGB-D camera; 3D human skeleton data can be generated based on the human body two-dimensional video stream and depth image stream; and the user's real-time input action sequence can be obtained based on the 3D human skeleton data.

[0107] The standard action sequence to be learned selected by the user can be directly obtained from the preset standard action database.

[0108] S22, inputting the acquired real-time input action sequence and standard action sequence into a human action evaluation model;

[0109] S23. The human motion evaluation model evaluates the similarity between the input motion sequence and the standard motion sequence, and outputs a feature score.

[0110] Traditional human motion evaluation methods use two independent steps, namely, human motion feature quantification and metric comparison, to evaluate human motion. Moreover, both steps require human experts to design, and the designed methods cannot meet the needs of all motions. There is always a large part of the motion that cannot be well recognized. Compared with this traditional human motion evaluation method, the human motion evaluation model based on the deep learning model in this application can couple feature quantification and metric comparison well, and has a higher recognition accuracy.

[0111] S3, using the sentiment analysis model to analyze the sentiment of users when viewing the feature scores output by the human action evaluation model, and constructing an error matching library based on the analysis results, such as Figure 5 As shown, the specific process is:

[0112] S31. Use an RGB-D camera to collect in real time the video stream of the user's expression when he sees the feature score output by the human action evaluation model, and send the collected video stream to the sentiment analysis model.

[0113] It should be noted that the sentiment analysis model can adopt the THIN in the prior art: THrowable information networks model, Deep FEA model or EmotionNet model.

[0114] S32. The sentiment analysis model analyzes the sentiment of the user when seeing the feature score output by the human motion evaluation model, and obtains the probability of the user's sentiment when seeing the feature score, wherein the sentiment includes happiness, confusion and frustration.

[0115] S33. When the probability of doubt among the three emotions is the largest, the sentiment analysis model compares the obtained doubt probability with the preset doubt probability threshold, and calculates the adjusted score based on the difference between the two and the adjustment base.

[0116] In order to more clearly understand how to calculate the adjusted score, a specific example is used below to illustrate.

[0117] The total probability of the user's emotion when seeing the feature rating obtained by the sentiment analysis model is 100%. Assume that the probability of being happy is 10%, the probability of being discouraged is 20%, the probability of being confused is 70%, the probability threshold of being confused is 50%, and the adjustment base is 10 points. The score that needs to be adjusted is: (70%-50%)*10=2, and the current adjusted score is: the last adjusted score + 2.

[0118] S34, taking the input action sequence and the standard action sequence corresponding to when the user's emotion is doubtful when seeing the feature score and the adjusted score as error samples, and using the error samples to build an error matching library.

[0119] For the sake of convenience, the input action sequence and standard action sequence corresponding to the user's emotion of doubt when seeing the feature score are called the wrong input action sequence and the wrong standard action sequence respectively. Specifically, the form of the error sample is [wrong standard action sequence, wrong input action sequence, adjusted score].

[0120] The human motion evaluation model can be used to obtain physically objective feature scores, and users have their own intuitive understanding of actions. The feature scores output by the human motion evaluation model are adjusted according to the output results of the sentiment analysis model, so that the adjusted feature scores take into account both objective evaluation and sensory evaluation, reducing the probability that traditional methods cannot recognize or have biased recognition, and making the final feature scores more objective.

[0121] S4, using the error matching library to update the human action evaluation model, such as Figure 6 As shown, the specific process is:

[0122] The samples in the mismatch library are input into the human action evaluation model.

[0123] The weights of the human motion evaluation model are updated through the back propagation algorithm, so that the loss function of the human motion evaluation model is reduced. When the loss function of the human motion evaluation model converges, the updating is stopped. The human motion evaluation model obtained at this time is the updated human motion evaluation model.

[0124] The human motion evaluation model is updated using the error matching library, so that the updated human motion evaluation model has the ability to recognize erroneous data. When encountering such erroneous data again, it can identify the evaluation results that meet the user's psychological expectations while taking into account the objective action essentials evaluation.

[0125] After each evaluation is completed, the human motion evaluation model will automatically update and optimize the model using the error samples in the error matching library without the help of external personnel, so that the next evaluation result of the human motion evaluation model will be more reasonable and accurate.

[0126] S5. Evaluate human motion using the updated human motion evaluation model.

[0127] The traditional human motion evaluation method is an open-loop evaluation method, which cannot supervise whether the score given by the model is appropriate. Compared with this traditional evaluation method, this application can make the evaluation of the human motion evaluation model a closed-loop evaluation by adding a sentiment analysis model, so that the feature score output by the human motion evaluation model can meet both the physical objective properties and the capabilities of human senses.

[0128] After being deployed on a device, the traditional human motion evaluation method is a solidified model that cannot self-repair errors. The model can only be manually upgraded when a command is manually sent to upgrade the model. Compared with this traditional human motion evaluation method, this application adds a full-life learning adjustment process, which enables the model to automatically correct itself after each error, ensuring that the same error will not be generated next time the same error is encountered, allowing the device to maintain the ability of continuous self-learning and automatic model optimization throughout its service life.

[0129] Based on the human motion evaluation method provided in the present application, the present application also provides a human motion evaluation device, which includes a human motion evaluation model, a sentiment analysis model and an error matching library.

[0130] Among them, the human motion evaluation model is used to evaluate the similarity between the real-time input motion sequence of the user and the standard motion sequence selected by the user to learn, and output a feature score.

[0131] The sentiment analysis model is used to analyze the sentiment of users when viewing the feature scores output by the human action evaluation model, and to build an error matching library based on the analysis results.

[0132] The error matching library is used to update the human motion evaluation model.

[0133] In a specific embodiment, the human motion evaluation model is obtained by prior training and deployment. The specific process of obtaining the human motion evaluation model is as follows:

[0134] First, an action evaluation dataset is generated based on human skeleton data, which includes the following steps:

[0135] The RGB-D camera is used to obtain the two-dimensional video stream and depth image stream of the human body.

[0136] Generate 3D human skeleton data based on human 2D video stream and depth image stream.

[0137] The input action sequence is obtained according to the 3D human skeleton data.

[0138] Obtain standard action sequences from the standard action database.

[0139] The input action sequence and the standard action sequence are subjected to feature quantification and metric comparison to obtain feature scores.

[0140] A large number of human motion evaluation samples are obtained based on a large number of standard motion sequences and input motion sequences as well as feature scores corresponding to the standard motion sequences and the input motion sequences.

[0141] Each human motion evaluation sample is manually screened, and the evaluation samples that do not meet the preset rules are discarded. The motion evaluation dataset is obtained based on the retained evaluation samples that meet the preset rules.

[0142] Secondly, the deep learning model is trained using the human motion evaluation samples in the motion evaluation dataset to obtain a human motion evaluation model.

[0143] In a specific embodiment, the sentiment analysis model analyzes the sentiment of the user when viewing the feature scores output by the human action evaluation model, and the specific process of building the error matching library according to the analysis results is as follows:

[0144] An RGB-D camera is used to collect in real time the facial expression video stream of the user when he sees the feature score output by the human action evaluation model corresponding to the action sequence input by the user in real time, and the collected video stream is sent to the sentiment analysis model.

[0145] The sentiment analysis model analyzes the user's emotions when seeing the feature scores output by the human motion evaluation model, and obtains the probability of the user's emotions when seeing the feature scores, where the emotions include happiness, confusion, and frustration; the sum of the probabilities of the three emotions is 1.

[0146] When the probability of doubt is the largest among the three emotions, the sentiment analysis model compares the obtained doubt probability with the preset doubt probability threshold, and calculates the adjusted score based on the difference between the two and the adjustment base.

[0147] The input action sequence and standard action sequence corresponding to when the user's emotion is doubtful when seeing the feature score, as well as the adjusted score, are used as error samples, and the error matching library is constructed using the error samples.

[0148] In a specific embodiment, the specific process of the error matching library updating the human motion evaluation model is as follows:

[0149] The error samples in the error matching library are input into the human action evaluation model.

[0150] The weights of the human motion evaluation model are updated through the back propagation algorithm, so that the loss function of the human motion evaluation model is reduced. When the loss function of the human motion evaluation model converges, the updating is stopped. The human motion evaluation model obtained at this time is the updated human motion evaluation model.

[0151] In the above-mentioned embodiments, the human body motion evaluation device of the present application further includes a standard motion database, in which standard motion sequences are stored, and the standard motion database responds to a standard motion selection instruction input by a user and outputs a standard motion sequence.

[0152] In another embodiment, the present application also provides a human motion evaluation system, which includes an RGB-D camera, a human-computer interaction interface, and a human motion evaluation device.

[0153] Among them, the RGB-D camera is used to collect two-dimensional video streams and depth image streams of the human body, and is also used to collect expression video streams when the user sees the feature scores output by the human motion evaluation model in the human motion evaluation device.

[0154] The user can select the standard action sequence to be learned through the human-computer interaction interface, and the human-computer interaction interface can also present the feature scores output by the human action evaluation model to the user.

[0155] The human motion evaluation device is used to construct an error matching library according to the analysis results of the sentiment analysis model, and use the error matching library to update the human motion evaluation model, and use the updated human motion evaluation model to evaluate the user's real-time input motion.

[0156] This application constructs a human motion evaluation method and evaluation system that can continuously self-learn from human emotions, maintain model optimization and self-update by connecting the process of obtaining the action evaluation data set through data annotation, the process of building a human motion evaluation model, the sentiment analysis model process and the full-life learning and updating process of the human motion evaluation model.

[0157] The above-mentioned embodiments of the present application can be implemented in various hardware, software coding or a combination of the two. For example, the embodiments of the present application can also represent the program code for executing the above-mentioned method in a data signal processor. The present application can also relate to the various functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array. The above-mentioned processor can be configured according to the present application to perform specific tasks, which are completed by executing machine-readable software code or firmware code that defines the specific method disclosed by the present application. The software code or firmware code development can represent different programming languages ​​and different formats or forms. Different target platform compiled software codes can also be represented. However, the different code styles, types and languages ​​of the software code for performing tasks according to the present application and other types of configuration codes do not depart from the spirit and scope of the present application.

[0158] The above description is only an illustrative specific implementation manner of the present application. Without departing from the concept and principle of the present application, any equivalent changes and modifications made by any technician in the field should fall within the scope of protection of the present application.

Claims

1. A human motion evaluation method, characterized in that: The following steps are involved: Train and deploy a human motion evaluation model; The human motion evaluation model is used to evaluate the user's human motion and obtain a feature score; The sentiment analysis model is used to analyze the user's sentiment when viewing the feature scores output by the human action evaluation model, and a wrong matching library is built based on the analysis results. The process is as follows: Use an RGB-D camera to collect the video stream of the user's expression when seeing the feature score output by the human action evaluation model in real time, and send the collected video stream to the sentiment analysis model; The sentiment analysis model analyzes the emotions of the user when seeing the feature scores output by the human action evaluation model, and obtains the probability of the user's emotions when seeing the feature scores; the emotions include happiness, confusion, and frustration; When doubt has the highest probability among the three emotions, the sentiment analysis model compares the obtained doubt probability with the preset doubt probability threshold, and calculates the adjusted score based on the difference between the two and the adjustment base; The input action sequence and standard action sequence corresponding to the user's doubtful emotion when seeing the feature score, as well as the adjusted score, are used as error samples, and the error matching library is constructed using the error samples; The human motion evaluation model is updated using the error matching library; The updated human motion evaluation model is used to evaluate human motion.

2. The human body motion evaluation method according to claim 1, characterized in that: The specific process of training and deploying the human motion evaluation model is as follows: The process of generating an action evaluation dataset based on human skeleton data is as follows: Use RGB-D camera to obtain human body two-dimensional video stream and depth image stream; Generate 3D human skeleton data based on human 2D video stream and depth image stream; Obtain input action sequence according to 3D human skeleton data; Perform feature quantification and measurement comparison on the input action sequence and the standard action sequence to obtain feature scores; The standard action sequence is obtained from a standard action database; According to the standard action sequence and the input action sequence and the feature scores corresponding to the standard action sequence and the input action sequence, a human action evaluation sample is obtained; Screening each human action evaluation sample to obtain an action evaluation data set; The deep learning model is trained using human motion evaluation samples in the motion evaluation dataset to obtain and deploy the human motion evaluation model.

3. The human body motion evaluation method according to claim 1, characterized in that: The specific process of evaluating the user's human body movements using the human body movement evaluation model to obtain feature scores is as follows: Obtaining the user's real-time input action sequence and the standard action sequence selected by the user to be learned; Inputting the acquired real-time input action sequence and standard action sequence into the human action evaluation model; The human motion evaluation model evaluates the similarity between the input motion sequence and the standard motion sequence and outputs a feature score.

4. The human body motion evaluation method according to claim 1, characterized in that: The specific process of updating the human motion evaluation model using the error matching library is as follows: Input the samples in the wrong matching library into the human action evaluation model; The weights of the human motion evaluation model are updated through the back propagation algorithm so that the loss function of the human motion evaluation model is reduced. When the loss function of the human motion evaluation model converges, the updating is stopped and the obtained human motion evaluation model is used as the updated human motion evaluation model.

5. A human motion evaluation device, characterized in that: Includes human action evaluation model, sentiment analysis model and error matching library; The human motion evaluation model is used to evaluate the similarity between the acquired real-time input motion sequence of the user and the standard motion sequence selected by the user to be learned, and output a feature score; The sentiment analysis model is used to analyze the sentiment of the user when viewing the feature scores output by the human action evaluation model, and to build an error matching library based on the analysis results, and the process is as follows: Use an RGB-D camera to collect the video stream of the user's expression when seeing the feature score output by the human action evaluation model in real time, and send the collected video stream to the sentiment analysis model; The sentiment analysis model analyzes the emotions of the user when seeing the feature scores output by the human action evaluation model, and obtains the probability of the user's emotions when seeing the feature scores; the emotions include happiness, confusion, and frustration; When doubt has the highest probability among the three emotions, the sentiment analysis model compares the obtained doubt probability with the preset doubt probability threshold, and calculates the adjusted score based on the difference between the two and the adjustment base; The input action sequence and standard action sequence corresponding to the user's doubtful emotion when seeing the feature score, as well as the adjusted score, are used as error samples, and the error matching library is constructed using the error samples; The error matching library is used to update the human motion evaluation model.

6. The human body motion evaluation device according to claim 5, characterized in that: The human motion evaluation model is obtained by prior training and deployment; the specific process of obtaining the human motion evaluation model is as follows: Generating an action evaluation dataset based on human skeleton data includes the following steps: Use RGB-D camera to obtain human body two-dimensional video stream and depth image stream; Generate 3D human skeleton data based on human 2D video stream and depth image stream; Obtain input action sequence according to 3D human skeleton data; Obtain standard action sequences from a standard action database; Perform feature quantification and measurement comparison on the input action sequence and the standard action sequence to obtain feature scores; According to the standard action sequence and the input action sequence and the feature scores corresponding to the standard action sequence and the input action sequence, a human action evaluation sample is obtained; Screening each human action evaluation sample to obtain an action evaluation data set; The deep learning model is trained using the human motion evaluation samples in the motion evaluation dataset to obtain a human motion evaluation model.

7. The human body motion evaluation device according to claim 5, characterized in that: The specific process of the sentiment analysis model analyzing the sentiment of the user when viewing the feature scores output by the human action evaluation model and constructing the error matching library according to the analysis results is as follows: Use an RGB-D camera to collect the video stream of the user's expression when the user sees the feature score output by the human action evaluation model corresponding to the user's real-time input action sequence, and send the collected video stream to the sentiment analysis model; The sentiment analysis model analyzes the emotions of the user when seeing the feature scores output by the human action evaluation model, and obtains the probability of the user's emotions when seeing the feature scores, wherein the emotions include happiness, confusion, and frustration; When doubt has the highest probability among the three emotions, the sentiment analysis model compares the obtained doubt probability with the preset doubt probability threshold, and calculates the adjusted score based on the difference between the two and the adjustment base; The input action sequence and standard action sequence corresponding to when the user's emotion is doubtful when seeing the feature score, as well as the adjusted score, are used as error samples, and the error matching library is constructed using the error samples.

8. The human body motion evaluation device according to claim 7, characterized in that: The specific process of the error matching library updating the human action evaluation model is as follows: Input the samples in the wrong matching library into the human action evaluation model; The weights of the human motion evaluation model are updated through the back propagation algorithm so that the loss function of the human motion evaluation model is reduced. When the loss function of the human motion evaluation model converges, the updating is stopped and the obtained human motion evaluation model is used as the updated human motion evaluation model.

9. A human motion evaluation system, characterized in that: It includes RGB-D camera, human-computer interaction interface and human motion evaluation device; The RGB-D camera is used to collect a two-dimensional video stream and a depth image stream of a human body, and is also used to collect a video stream of an expression of a user when he sees a feature score output by a human action evaluation model in a human action evaluation device; The user selects a standard action sequence to be learned through the human-computer interaction interface, and the human-computer interaction interface presents the feature scores output by the human action evaluation model to the user; The human action evaluation device is used to build an error matching library according to the analysis results of the sentiment analysis model, and use the error matching library to update the human action evaluation model, and use the updated human action evaluation model to evaluate the user's real-time input action; The process of building an error matching library based on the analysis results of the sentiment analysis model is as follows: Use an RGB-D camera to collect the video stream of the user's expression when seeing the feature score output by the human action evaluation model in real time, and send the collected video stream to the sentiment analysis model; The sentiment analysis model analyzes the emotions of the user when seeing the feature scores output by the human action evaluation model, and obtains the probability of the user's emotions when seeing the feature scores; the emotions include happiness, confusion, and frustration; When doubt has the highest probability among the three emotions, the sentiment analysis model compares the obtained doubt probability with the preset doubt probability threshold, and calculates the adjusted score based on the difference between the two and the adjustment base; The input action sequence and standard action sequence corresponding to when the user's emotion is doubtful when seeing the feature score, as well as the adjusted score, are used as error samples, and the error matching library is constructed using the error samples.

Images