A method and system for action evaluation based on key frames and non-key frames

Abstract

The application discloses a kind of action evaluation method and system based on key frame and non-key frame, belong to machine vision field, method includes the three-dimensional coordinates of each frame of 32 human body joint points acquired by vision sensor export;Select feature joint;The abnormal three-dimensional coordinates of feature joint are corrected;Calculate the angle of each frame between feature joint in XOY, XOZ, YOZ three planes, draw angle change curve;If the curve is composed of multiple actions, it is segmented into single action curve;Actual action and its standard action curve are compared in different ways at key frame and non-key frame, the absolute value of angle difference at key frame is calculated, and the final evaluation result is obtained;System includes data import module, feature joint selection module, data processing module, angle calculation and curve drawing module, curve processing module and action evaluation module.The application has high accuracy, simple and efficient process, and the calculation process is simple and easy to understand.

Description

Technical Field

[0001] This invention relates to the field of machine vision, and in particular to a motion evaluation method and system based on keyframes and non-keyframes. Background Technology

[0002] In real life, there are many scenarios where the standard of movement needs to be assessed. For example, when conducting physical fitness tests on students, the movements of sit-ups, pull-ups, etc. need to be assessed; in physical education teaching, teachers need to correct and judge the movements of each student; in sports competitions, referees need to assess the movements of athletes; and in rehabilitation training, doctors will assess the movements of patients to detect their physical recovery.

[0003] With the development of computer technology, motion evaluation methods based on optimal frame selection have emerged in existing technologies. For example, Chinese invention patent CN114092862A, entitled "A Motion Evaluation Method Based on Optimal Frame Selection," describes a method that performs skeleton and pose analysis on a test subject's video to obtain a key point coordinate map. Based on the motion characteristics, the most representative video frames are selected and compared with standard template video frames to calculate the similarity, thereby obtaining the evaluation result.

[0004] The aforementioned invention, in an effort to reduce computation, evaluates actions solely by calculating the similarity of keyframes, ignoring the comparison of non-keyframes, thus reducing the accuracy of the evaluation results. Furthermore, when calculating similarity, the invention also determines whether the joint rotates clockwise or counterclockwise, and by how many degrees. Calculating the rotation angle and determining clockwise / counterclockwise rotation are relatively complex, increasing the difficulty of the evaluation and consequently reducing its efficiency. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a motion evaluation method and system based on key frames and non-key frames, so as to solve the problem of not ignoring the comparison of non-key frames in the video with less computation, so as to make the evaluation results more accurate; and to use a simpler and more efficient method to calculate motion similarity.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A motion evaluation method based on keyframes and non-keyframes includes the following steps:

[0008] Step 1: Export the 3D coordinates of each frame of the 32 joints of the human body acquired by the visual sensor;

[0009] Step 2, select the key joints;

[0010] Step 3: Correct the abnormal three-dimensional coordinates of the feature joints;

[0011] Step 4: Calculate the angles between feature joints in each frame on the XOY, XOZ, and YOZ planes, and plot the change curves.

[0012] Step 5: If the curve consists of multiple actions, it needs to be divided into curves for individual actions.

[0013] Step 6: Compare the curves of the actual movement with those of the standard movement, calculate the absolute value of the angle difference, and obtain the final evaluation result.

[0014] A further improvement of the technical solution of the present invention is that, in step 2, when selecting characteristic joints, based on the characteristics of the action itself, joints that can express the characteristics and key points of the action and can be conveniently evaluated are selected from among the 32 skeletal joints.

[0015] A further improvement to the technical solution of this invention is as follows: In step 3, since the visual sensor inevitably produces some coordinate anomalies when acquiring the three-dimensional coordinates of the skeletal joints, it is necessary to correct the abnormal coordinates. The correction method is as follows: if both sides of the abnormal data are normal data, the average value of the normal data closest to both sides is taken during correction; if several abnormal data appear one after another, the average value of the closest normal data on both sides is taken when correcting the first abnormal data, the average value of the corrected value of the first abnormal data and the average value of the closest normal data after it is taken when correcting the second abnormal data, the average value of the corrected value of the second abnormal data and the average value of the closest normal data after it is taken when correcting the third abnormal data, and so on, until all consecutive abnormal data are corrected.

[0016] A further improvement of the technical solution of the present invention is that: in step 4, the angle formed by each frame of the feature joints in the three planes of XOY, XOZ, and YOZ is calculated, where the feature joints refer to the feature joints that can represent a certain part and are associated with the corresponding angles of these feature joints.

[0017] A further improvement to the technical solution of this invention is as follows: In step 4, the angles formed by each frame of the feature joints in the three planes XOY, XOZ, and YOZ are calculated. To calculate the angles formed between several joints, it is necessary to find their rotation axes, and then obtain the three-dimensional coordinates with the rotation axis as the origin. Then, based on the new three-dimensional coordinates, the angles formed by the joints in the three planes and the corresponding coordinate axes are calculated. The corresponding coordinate axes are the positive X-axis when in the XOY plane, the positive X-axis when in the XOZ plane, and the positive Y-axis when in the YOZ plane. Finally, the angles are subtracted to obtain the included angles between these feature joints. The arctangent function is used to calculate the angles; the angle changes in the three planes XOY, XOZ, and YOZ are calculated.

[0018] In the XOY plane, using the formula Calculate the change in angle;

[0019] In the XOZ plane, using the formula Calculate the change in angle;

[0020] In the YOZ plane, using the formula Calculate the change in angle;

[0021] use When calculating the angle, it is necessary to consider the specific quadrant in which its (A,B) coordinates lie. That is, when A>0, the angle can be calculated based on... The actual angle is obtained directly. When A < 0, the actual angle is...

[0022] A further improvement of the technical solution of the present invention is that, in step 5, based on the characteristics of the action itself, the angle change curve composed of multiple actions is segmented to separate the angle change curves of each action into the angle change curve of a single action.

[0023] A further improvement of the technical solution of the present invention is that: in step 6, based on the characteristics of the action itself and the characteristics of the curve change, the location of the key frame can be determined more accurately; the angle change curves of the standard action feature joints projected onto the three planes XOY, XOZ, and YOZ are compared with the angle change curves corresponding to the actual action.

[0024] The specific steps are as follows:

[0025] 6.1 Determine whether there is a curve in the standard motion angle change curve that can be regarded as a horizontal straight line;

[0026] If there is a curve in the standard motion angle change curve that can be regarded as a horizontal straight line, find the curve in the actual motion angle change curve that has the same feature joint point and plane. First, determine whether the actual motion angle change curve can also be regarded as a horizontal straight line. If not, directly determine that the motion is unqualified and do not need to perform any further judgments. If yes, directly calculate the absolute value of the angle difference between the angle change curve corresponding to the standard motion at the keyframe.

[0027] If there is no curve in the standard motion angle change curve that can be regarded as a horizontal straight line, proceed directly to the next step;

[0028] 6.2 For the standard motion angle change curve, it cannot be regarded as a horizontal straight line. First, compare the angle change trend of the non-keyframe part of the actual motion angle change curve. If its angle change trend is roughly similar to the angle change curve corresponding to the standard motion, then only the keyframe part needs to be compared, and the absolute value of the angle difference between the keyframe and the standard motion is calculated. If its angle change trend is inconsistent with the angle change curve corresponding to the standard motion, it is directly judged as an unqualified motion, and there is no need to compare the keyframe part.

[0029] 6.3 Calculate the sum of the absolute values ​​of all angle differences; the smaller the sum, the more standard the movement.

[0030] By combining the angle changes of non-keyframe parts, the angle change trends of non-keyframe parts are first compared, and then the keyframe parts are compared. When the non-keyframe parts meet the standards, the results are completely dominated by the keyframe parts. If the non-keyframe parts do not meet the standards, the evaluation results are completely determined by the non-keyframe parts.

[0031] A motion evaluation system based on keyframes and non-keyframes includes a data import module, a feature joint selection module, a data processing module, an angle calculation and curve drawing module, a curve processing module, and a motion evaluation module.

[0032] The data import module can import the three-dimensional coordinates of each frame of the 32 skeletal joints of the human body obtained by the visual sensor into the computer.

[0033] The feature joint selection module, while ensuring the accuracy of motion evaluation, selects some joints that can represent the motion from 32 skeletal joints based on the motion characteristics, referred to as feature joints.

[0034] In order to prevent individual coordinate data anomalies from affecting the final evaluation result when the visual sensor collects the three-dimensional coordinates of human skeletal joints, the data processing module needs to correct the abnormal three-dimensional coordinates of the feature joints.

[0035] The angle calculation and curve drawing module calculates the angles formed between feature joints in each frame on the three planes XOY, XOZ, and YOZ, and draws the angle change curves.

[0036] If the obtained angle change curve is a curve of an action performed many times consecutively or a curve of multiple actions performed consecutively, the curve processing module needs to divide the curve into angle change curves of a single action; if the obtained curve is an angle change curve of a single action, this module is skipped directly.

[0037] The motion evaluation module compares the trends of the actual motion angle change curve with the standard motion angle change curve at non-keyframes; at keyframes, it compares the angles of the actual motion with the standard motion; and combines these two parts to obtain the final evaluation result.

[0038] The technological advancements achieved by this invention due to the adoption of the above technical solutions are as follows:

[0039] 1. This invention projects the three-dimensional coordinates of feature joints onto three planes: XOY, XOZ, and YOZ, and calculates the angles of each frame on these three planes. Because obtaining the changes in joint angles projected onto the three planes allows for a very accurate description of the action, this calculation method is simpler and more efficient.

[0040] 2. This invention combines keyframes and non-keyframes for evaluation, resulting in more standardized evaluation results. If a non-keyframe does not meet the standard, the evaluation result is entirely determined by the non-keyframe portion, and the result is deemed unqualified. When the non-keyframe portion meets the standard, the result is entirely dominated by the keyframe portion; only the absolute value of the angle difference between the actual action keyframe and the standard action keyframe needs to be calculated. The similarity is then obtained by summing all the absolute values. This simplifies the calculation process. Therefore, this evaluation method is more accurate and has a simpler calculation process. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the markings of 32 skeletal joints of the human body and the corresponding joint numbers obtained by the Azure Kinect camera in an embodiment of the present invention;

[0042] Figure 2 This is an example diagram of the three-dimensional coordinates of one of the skeletal joint points acquired by the Azure Kinect camera in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the three-dimensional coordinate orientation of the Azure Kinect camera in an embodiment of the present invention;

[0044] Figure 4 This is a flowchart of an action evaluation method based on keyframes and non-keyframes in an embodiment of the present invention;

[0045] Figure 5 This is a flowchart of the evaluation process for the angle change curve of the action in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of an action evaluation system based on keyframes and non-keyframes in an embodiment of the present invention;

[0047] Figure 7 This is a schematic diagram of a sit-up in an embodiment of the present invention;

[0048] Figure 8 This is a graph showing the angle changes of the spine relative to the knee in three planes with the pelvis as the axis of rotation in an embodiment of the present invention.

[0049] Figure 9 This is a graph showing the angular changes of the ankle, knee, and hip in three planes in an embodiment of the present invention;

[0050] Figure 10 This is the first sit-up curve in this invention, showing the change in the angle of the spine relative to the knee with the pelvis as the axis of rotation.

[0051] Figure 11 This is a graph showing the change in the angle of the spine relative to the knee with the pelvis as the axis of rotation during the second sit-up in this embodiment of the invention.

[0052] Figure 12 This is the first embodiment of the invention showing the angular variation curves of the ankle, knee, and hip in three planes during a sit-up.

[0053] Figure 13 This is a graph showing the angle changes of the ankle, knee, and hip in three planes during a second sit-up in this embodiment of the invention. Detailed Implementation

[0054] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments:

[0056] The Azure Kinect camera consists of a depth camera, an RGB color camera, an IR infrared transmitter, and a 7-microphone array. By selecting the appropriate mode and having the person positioned within the appropriate field of view of the Azure Kinect camera, images can be captured. Figure 1 The template contains the three-dimensional coordinates of 32 skeletal joints of the human body. The names of the skeletal joints corresponding to each number are shown in Table 1-1. Figure 2 As shown, the number after index represents the number of the bone joint; the data after cameraX represents the distance of this joint relative to the origin in the X-axis direction; the data after cameraY represents the distance of this joint relative to the origin in the Y-axis direction; and the data after cameraZ represents the distance of this joint relative to the origin in the Z-axis direction. Figure 3This is a schematic diagram showing the specific directions of the three-dimensional coordinates of an Azure Kinect camera.

[0057] Table 1-1 Names of skeletal joints corresponding to each number

[0058] serial number name serial number name serial number name 0 pelvis 1 Spine 2 Thoracic cavity 3 neck 4 Left clavicle 5 left shoulder 6 left elbow 7 left wrist 8 left palm 9 left fingertips 10 left thumb 11 Right clavicle 12 right shoulder 13 right elbow 14 right wrist 15 right palm 16 right fingertips 17 Right thumb 18 Left hip 19 Left knee 20 left ankle 21 left foot 22 Right hip 23 Right knee 24 right ankle 25 Right foot 26 jaw 27 nose 28 Left eye 29 left ear 30 right eye 31 right ear

[0059] like Figure 4 As shown, an action evaluation method based on keyframes and non-keyframes includes:

[0060] Step 1: Export the 3D coordinates of each frame of the 32 joints of the human body acquired by the visual sensor;

[0061] The Azure Kinect visual sensor is used to acquire information about the human skeletal joints. The Azure Kinect records the human's movement, and then the 3D coordinates of each frame of the 32 skeletal joints captured by the Azure Kinect camera are imported into a computer. Installing the corresponding Azure Kinect API program on the computer allows you to import these 3D coordinates (detailed steps can be found on the Microsoft website: https: / / docs.microsoft.com / zh-cn / azure / kinect-dk / ). The Azure Kinect camera typically acquires 30 frames per second.

[0062] Step 2, select the key joints;

[0063] To ensure the accuracy of motion assessment, and to improve the efficiency of motion assessment and simplify subsequent calculations, it is not necessary to use all 32 skeletal joints. Instead, it is only necessary to select some joints that can represent the motion based on its characteristics, referred to as characteristic joints.

[0064] Step 3: Correct the abnormal three-dimensional coordinates of the feature joints;

[0065] To prevent individual coordinate anomalies from affecting the final evaluation results when the Azure Kinect camera acquires the 3D coordinates of human skeletal joints, it is necessary to correct the 3D coordinates of abnormal feature joints. In this invention, if both sides of an abnormal data point are normal data, the average of the two closest normal data points is used for correction. If several abnormal data points appear consecutively, the correction for the first abnormal data point is the average of the two closest normal data points, the correction for the second abnormal data point is the average of the corrected value of the first abnormal data point and the next closest normal data point, the correction for the third abnormal data point is the average of the corrected value of the second abnormal data point and the next closest normal data point, and so on, until all consecutive abnormal data points are corrected. For example, consider the data 1.2, 1.3, 1.4, 1.5, 3, 5, 6, 1.6, 1.4, 1.3, 1.2. Among these data, 3, 5, and 6 are abnormal data points. When correcting the first abnormal data point 3, the average of its two closest normal data points 1.5 and 1.6, which is 1.55, is used. When correcting the second outlier 5, the average of the corrected value of the first outlier (1.55) and the nearest normal data point (1.6) is taken, resulting in a value of 1.575. When correcting the third outlier 6, the average of the corrected value of the second outlier (1.575) and the nearest normal data point (1.6) is taken, resulting in a value of 1.5875.

[0066] Step 4: Calculate the angles between feature joints in each frame on the XOY, XOZ, and YOZ planes, and plot the change curves.

[0067] Here, "feature joints" refers to the number of feature joints that represent a specific part and whose angles correspond to each other. To calculate the angles formed between several joints, it is necessary to find their rotation axes, obtain their 3D coordinates with the rotation axes as the origin, and then calculate the angles formed by the joints in the three planes with their corresponding coordinate axes (corresponding coordinate axes are the positive X-axis in the XOY plane, the positive X-axis in the XOZ plane, and the positive Y-axis in the YOZ plane). Finally, the angles in the same frame on the same plane are subtracted to obtain the included angles between these feature joints in the three planes. For example, if there are three feature joints A, B, and C, the angle changes between them can represent the angle changes of a certain part, where B is the rotation axis of joints A and C. To calculate the angle between three joints A, B, and C, first, using the feature joint B as the origin, recalculate the 3D coordinates of A and C with B as the origin. Then, project the new 3D coordinates of A and C onto the XOY, XOZ, and YOZ planes respectively. Next, calculate the angles formed by joints A and C in each frame on their corresponding coordinate axes in these three planes. Finally, subtract the angles of joints A and C in the same frame on the same plane to obtain the angle difference between A and C. Plot the curves showing the change of the angle difference between A and C on each plane. The horizontal axis represents the frame, and the vertical axis represents the angle.

[0068] In this invention, the arctangent function is used to calculate the angles. The angular changes in the XOY, XOZ, and YOZ planes are calculated. In the XOY plane, the formula is used... Calculate the angle change. In the XOZ plane, use the formula... Calculate the angle change. In the YOZ plane, use the formula... Calculate the angle change. (Note: Use...) When calculating the angle, it is necessary to consider the specific quadrant in which its (A,B) coordinates lie. That is, when A>0, it can be calculated based on... The actual angle is obtained directly. When A < 0, the actual angle is...

[0069] When assessing the movement of different people, although each person has different body characteristics, when different people perform the same movement, as long as the movement is standard, the angle changes between their joints will be very similar. Therefore, using angle change curves for movement assessment will be more accurate.

[0070] The angular changes between key points on three planes can fully reflect the changes in motion. Using the coordinates projected onto the three planes and the arctangent trigonometric function, the angle can be easily calculated. Compared to calculating rotation angles and determining clockwise or counterclockwise rotation, the calculation process is simple and efficient.

[0071] Step 5: If the curve consists of multiple actions, it needs to be divided into curves for individual actions.

[0072] Since this invention evaluates a single action, it is necessary to obtain the angle change curve of that single action. If the obtained angle change curve represents a single action performed many times consecutively or multiple actions performed consecutively, the curve needs to be segmented into angle change curves for each individual action. If the obtained curve is simply the angle change curve for a single action, this step can be omitted.

[0073] Step 6: Compare the curves of the actual movement with those of the standard movement, calculate the absolute value of the angle difference, and obtain the final evaluation result.

[0074] By combining the characteristics of the movement itself and the changes in the curves, the location of keyframes can be determined relatively accurately (for example, in a sit-up, based on the characteristics of the sit-up movement, the keyframes can be determined at the beginning of the sit-up, when the angle between the upper and lower body reaches its maximum, and when the sit-up ends). The angle change curves between the characteristic joints of the standard movement in the XOY, XOZ, and YOZ planes are compared with the corresponding angle change curves of the actual movement. The flowchart is as follows... Figure 5 As shown, the steps are as follows:

[0075] 6.1 Determine whether there is a curve in the standard motion angle change curve that can be regarded as a horizontal straight line.

[0076] If the angle change curve of the standard motion contains a curve that can be considered a horizontal straight line, find the actual motion angle change curve that shares the same feature joint point and plane. First, determine if the actual motion angle change curve can also be considered a horizontal straight line. If not, directly determine that the motion is unqualified, without needing to perform any further checks. If it can, directly calculate the absolute value of the angle difference between the angle change curve corresponding to the standard motion and the keyframe.

[0077] If there is no curve in the standard motion angle change curve that can be regarded as a horizontal straight line, proceed directly to the next step.

[0078] 6.2 The standard motion angle change curve cannot be considered a horizontal straight line. First, compare the angle change trend of the non-keyframe portion of the actual motion angle change curve. If its angle change trend is roughly similar to the angle change curve corresponding to the standard motion, then only the keyframe portion needs to be compared, and the absolute value of the angle difference between it and the keyframe of the standard motion needs to be calculated. If its angle change trend is inconsistent with the trend of the angle change curve corresponding to the standard motion, then it is directly judged as an unqualified motion, and there is no need to compare the keyframe portion.

[0079] 6.3 Calculate the sum of the absolute values ​​of all angle differences. The smaller the sum, the more standard the movement.

[0080] For motion joints, this invention combines the angle changes of keyframes and non-keyframes. First, it compares the angle change trends of non-keyframes, then compares the keyframes. When the non-keyframes meet the standard, the evaluation result is entirely determined by the keyframes; if the non-keyframes do not meet the standard, the evaluation result is entirely determined by the non-keyframes. This method is more accurate than methods that almost completely ignore non-keyframes. Ignoring non-keyframes and only considering keyframes can easily lead to the actual motion differing completely from the standard motion in the non-keyframes, severely impacting the accuracy of motion evaluation. The calculation process is also simpler. If the non-keyframes do not meet the standard, the evaluation result is entirely determined by the non-keyframes and judged as unqualified. When the non-keyframes meet the standard, the result is entirely determined by the keyframes; only the absolute value of the angle difference between the actual motion keyframe and the standard motion keyframe needs to be calculated. The similarity score is then obtained by summing all the calculated absolute values.

[0081] like Figure 6 As shown, a motion evaluation system based on keyframes and non-keyframes is disclosed. The system includes a data import module, a feature joint selection module, a data processing module, an angle calculation and curve drawing module, a curve processing module, and a motion evaluation module.

[0082] The data import module imports the three-dimensional coordinates of each frame of the 32 skeletal joints of the human body obtained by the visual sensor into the computer.

[0083] The feature joint selection module: On the basis of ensuring the accuracy of motion evaluation, it selects some joints that can represent the motion from 32 skeletal joints according to the characteristics of the motion, referred to as feature joints.

[0084] The data processing module: In order to prevent individual coordinate data anomalies from affecting the final evaluation result when the vision sensor collects the three-dimensional coordinates of human skeletal joints, it is necessary to correct the abnormal three-dimensional coordinates of the feature joints.

[0085] The angle calculation and curve drawing module calculates the angles formed between feature joints in each frame on the three planes XOY, XOZ, and YOZ, and draws the angle change curves.

[0086] The curve processing module: If the obtained angle change curve is a curve representing a continuous action performed many times, or a curve representing multiple actions performed consecutively, the curve needs to be segmented into angle change curves for individual actions. If the obtained curve is simply the angle change curve for a single action, this module is skipped.

[0087] The motion evaluation module compares the trends of the actual motion angle change curve with the standard motion angle change curve at non-keyframes. At keyframes, it compares the angles of the actual motion with those of the standard motion. Combining these two parts yields the final evaluation result.

[0088] Example

[0089] Taking sit-ups as an example, this paper elaborates on evaluation methods and systems based on keyframes and non-keyframes.

[0090] Step 1: Import the 3D coordinates of each frame of the 32 skeletal joints of the human body captured by the Azure Kinect camera into the computer.

[0091] Step 2: When doing sit-ups, begin by lying supine on a mat with your legs straight and together, feet flat on the ground, back pressed against the ground, and hands gently placed at your sides; Figure 7 As shown. When sitting up, the head and upper body leave the ground, with the upper body forming a 90-degree angle with the legs as much as possible. The buttocks should not leave the ground, while the lower body remains almost unchanged. When the body and legs are at approximately a 90-degree angle, the upper body returns to the supine position. The key parts of the sit-up are keeping the legs straight and together, feet flat on the ground, and the angle between the upper body and legs. The upper body movement can be replaced by the movement of the spine around the pelvis, while the lower body only needs to maintain the position of the legs straight and together with the feet flat on the ground. Therefore, the ankles, knees, buttocks, pelvis, and spine can be selected as characteristic joint points. The three-dimensional coordinates of the pelvis are listed in Table 1-2.

[0092] Table 1-2 Three-dimensional coordinates of the pelvis (unit: millimeters)

[0093]

[0094]

[0095] The three-dimensional coordinates of the spine are listed in Table 1-3.

[0096] Table 1-3 Three-dimensional coordinates of the spine (unit: millimeters)

[0097]

[0098]

[0099] The three-dimensional coordinates of the left hip, left knee, and left ankle are shown in Tables 1-4, 1-5, and 1-6, respectively.

[0100] Table 1-4 Three-dimensional coordinates of the left hip (unit: millimeters)

[0101]

[0102]

[0103] Table 1-5 Three-dimensional coordinates of the left knee (unit: millimeters)

[0104]

[0105]

[0106] Table 1-6 Three-dimensional coordinates of the left ankle (unit: millimeters)

[0107]

[0108]

[0109] Step 3: To prevent individual coordinate anomalies from affecting the final evaluation results when the Azure Kinect camera acquires the 3D coordinates of human skeletal joints, it is necessary to correct the 3D coordinates of abnormal feature joints. In this invention, if both sides of an abnormal data point are normal data, the average of the two closest normal data points is used for correction. If several abnormal data points appear consecutively, the correction for the first abnormal data point is the average of the two closest normal data points; the correction for the second abnormal data point is the average of the corrected value of the first abnormal data point and the next closest normal data point; the correction for the third abnormal data point is the average of the corrected value of the second abnormal data point and the next closest normal data point, and so on, until all consecutive abnormal data points are corrected. In this example, at frame 27, there is an abnormal data point -697.2325439 on the X-axis of the left hip, which is corrected to the average of -642.8029785 and -634.0131226, resulting in -638.40805055. The corrected values ​​are then used in subsequent angle calculations.

[0110] Step 4: Since the angle change of the spine relative to the knee with the pelvis as the axis of rotation during sit-ups reflects the overall movement of the upper body, while the lower body remains almost still, the relationship between the three joints—pelvis, spine, and knee—can reflect the movement of the upper body relative to the lower body. Therefore, the pelvis is chosen as the origin of the coordinate system for the spine and knee, and the three-dimensional coordinates of their characteristic joints are recalculated. The new three-dimensional coordinates of the spine and knee are projected onto the XOY, XOZ, and YOZ planes respectively. Then, the angles of the spine and knee relative to the pelvis in each frame are calculated in these three planes. The angle difference is obtained by subtracting the angle of the knee relative to the pelvis from the angle of the spine relative to the pelvis, and finally, the angle difference curves in each plane are plotted. Since the lower body only needs to keep the legs straight and together with the feet flat on the ground and remain still, the angle change of the legs can be replaced by the angle change between the ankle and hip with the knee as the axis of rotation. First, re-establish the three-dimensional coordinates of the ankle and hip using the knee as the origin. Then, calculate the angles formed by the ankle and hip with the corresponding coordinate axes in the three planes. Next, subtract these angles to obtain the angle differences. Finally, plot the angle variation curves.

[0111] Calculate the angular changes in the XOY, XOZ, and YOZ planes. In the XOY plane, use the formula... Calculate the angle change. In the XOZ plane, use the formula... Calculate the angle change. In the YOZ plane, use the formula... Calculate the angle change. (Note: Use...) When calculating the angle, it is necessary to consider the specific quadrant in which its (A,B) coordinates lie. That is, when A>0, it can be calculated based on... The actual angle is obtained directly. When A < 0, the actual angle is... The three-dimensional coordinates of the spine with the pelvis as the origin are shown in Table 1-7.

[0112] Table 1-7 Three-dimensional coordinates of the spine with the pelvis as the origin (unit: millimeters)

[0113]

[0114]

[0115] The curves showing the angular changes of the spine relative to the knee in three planes with the pelvis as the axis of rotation, and the curves showing the angular changes of the ankle, knee, and hip in three planes are as follows: Figure 8 , 9 As shown.

[0116] Step 5: Since this invention evaluates a single action, it is necessary to obtain the angle change curve of that single action. If the obtained angle change curve represents a single action performed many times consecutively, or multiple actions performed consecutively, the curve needs to be segmented into individual action angle change curves. Based on the characteristics of the obtained angle change curve and the sit-up movement itself, in this example, individual action segmentation can be performed according to the pattern of peaks and troughs in the curve. The segmentation result is as follows: Figure 10 , 11 As shown in Figures 12 and 13.

[0117] Step 6: Compare the angle change curves of the feature joints projected onto the XOY, XOZ, and YOZ planes with the angle change curves of a standard sit-up. Since the Azure Kinect camera typically acquires 30 frames per second, the horizontal axis represents the frame count; more frames mean a longer time. The reason it's not necessary to compare the curve trends very closely is that everyone's sit-up speed is different, and the time spent on each stage of a sit-up also varies, naturally resulting in different rates of ascent and descent of the angle change curves.

[0118] Based on the characteristics of the motion itself and the changes in the curve, the location of the keyframe can be determined relatively accurately.

[0119] When the standard motion angle change curve can be considered a horizontal curve, firstly, directly compare it with the corresponding angle change curve of the actual motion. If it cannot be considered a horizontal curve (i.e., the non-keyframe portion is unacceptable), then it is directly judged as an unacceptable motion, and no further comparison is needed. If it can be considered a horizontal curve (i.e., the non-keyframe portion is acceptable), then only the angles at the keyframes of the corresponding curves need to be compared, and the absolute value of the difference between the two angles needs to be calculated. Secondly, compare other curves. When judging other curves, first compare whether the upward or downward trends of the curves are roughly consistent, i.e., the trend of the curve change in the non-keyframe portion. If they are inconsistent, then it is directly judged as an unacceptable motion, and no further comparison is needed. If they are acceptable, then directly compare the angles at the keyframes of the corresponding curves and calculate the absolute value of the angle difference. Finally, calculate the sum of the absolute values ​​of all angle differences. The smaller the sum, the closer it is to the standard motion, that is, the more standard the motion is.

[0120] Assuming the second sit-up is the standard sit-up, compare the first sit-up with the standard sit-up (the second sit-up).

[0121] Based on the characteristics of the sit-up movement, keyframes can be determined at the beginning of the sit-up, when the angle between the upper and lower body reaches its maximum, and when the sit-up ends. Furthermore, the characteristics of the curve can help us further pinpoint the specific location of the keyframes.

[0122] Based on the angle change curve of the sit-up, the keyframes of the standard sit-up can be determined as follows: frame 1 (at the very beginning of the sit-up), frame 26 (when the angle between the upper and lower body reaches its maximum), and frame 48 (at the end of the sit-up). The standard sit-up's angle change curves, with the pelvis as the axis of rotation, relative to the spine and knees / ankles, and with the knees and hips as the axis of rotation, are shown in the XOZ, XOY, and YOZ planes. The angles at the keyframes are as follows:

[0123] The angle formed by the spine relative to the knee with the pelvis as the axis of rotation:

[0124] In frame 1, the angle in the XOY plane is -147.0535. The angle in the XOZ plane is 154.4737. The angle in the YOZ plane is 0.76258.

[0125] At frame 26, the angle in the XOY plane is -164.3129. The angle in the XOZ plane is 233.7191. The angle in the YOZ plane is 55.5697.

[0126] At frame 48, the angle in the XOY plane is -154.7710. The angle in the XOZ plane is 164.5247. The angle in the YOZ plane is 2.6781.

[0127] The angles formed by the ankle, knee, and hip with the knee as the axis in the XOZ, XOY, and YOZ planes:

[0128] In frame 1, the angle in the XOY plane is -178.6919. The angle in the XOZ plane is 179.5167. The angle in the YOZ plane is 184.5603.

[0129] At frame 26, the angle in the XOY plane is -181.2275. The angle in the XOZ plane is 181.6719. The angle in the YOZ plane is 189.0480.

[0130] At frame 48, the angle in the XOY plane is -179.6072. The angle in the XOZ plane is 181.2622. The angle in the YOZ plane is 190.8554.

[0131] Next, the actual sit-up (i.e., the first sit-up) is evaluated. Based on the curve characteristics, the keyframes of the actual sit-up can be determined as follows: frame 1 (when the sit-up just begins), frame 28 (when the angle between the upper and lower body reaches its maximum), and frame 53 (when the sit-up ends).

[0132] First, comparing the horizontal curves, the angle changes of the ankles, knees, and hips in the XOZ and XOY planes during an actual sit-up can be considered as horizontal straight lines. Therefore, we only need to calculate the absolute value of the angle difference between the actual sit-up and the standard sit-up at the keyframe. The angles of the horizontal angle change curve of the actual sit-up at the keyframe are as follows:

[0133] In frame 1, the angle in the XOY plane is -179.7602. The angle in the XOZ plane is 180.4639.

[0134] At frame 28, the angle in the XOY plane is -179.2546. The angle in the XOZ plane is 181.2736.

[0135] At frame 53, the angle in the XOY plane is -179.7425. The angle in the XOZ plane is 180.0344.

[0136] Calculate the absolute value of the angle difference between the actual sit-up and the standard sit-up at the corresponding keyframes:

[0137] The absolute values ​​of the angle differences between the first frame of the actual sit-up and the first frame of the standard sit-up in the XOY and XOZ planes are -178.6919-(-179.7602) and 180.4639-179.5167, respectively, which are 1.0683 and 0.9472.

[0138] The absolute values ​​of the angle differences between the actual sit-up frame 28 and the standard sit-up frame 26 in the XOY and XOZ planes are -179.2546-(-181.2275) and 181.6719-181.2736, respectively, with results of 1.9729 and 0.3983.

[0139] The absolute values ​​of the angle differences between the actual sit-up frame 53 and the standard sit-up frame 48 in the XOY and XOZ planes are -179.6072-(-179.7425) and 181.2622-180.0344, respectively, with results of 0.1353 and 1.2278.

[0140] Secondly, by comparing the other curves, it can be seen from the above curves that their rising and falling trends are roughly consistent, so it is only necessary to compare the angles of the keyframes.

[0141] The actual sit-up curve at the keyframe angle is as follows:

[0142] The angle formed by the spine relative to the knee with the pelvis as the axis of rotation:

[0143] In frame 1, the angle in the XOY plane is -151.5323. The angle in the XOZ plane is 159.4893. The angle in the YOZ plane is -3.4868.

[0144] At frame 28, the angle in the XOY plane is -161.7457. The angle in the XOZ plane is 197.8144. The angle in the YOZ plane is 56.1036.

[0145] At frame 53, the angle in the XOY plane is -149.5006. The angle in the XOZ plane is 157.6072. The angle in the YOZ plane is -3.8555.

[0146] Angles of the ankle, knee, and hip at keyframes in the YOZ plane with the knee as the axis:

[0147] In the first frame, the angle in the YOZ plane is 185.9944.

[0148] At frame 28, the angle in the YOZ plane is 192.0269.

[0149] At frame 53, the angle in the YOZ plane is 182.0287.

[0150] Calculate the absolute value of the angle difference between the actual sit-up and the standard sit-up at the corresponding keyframes:

[0151] The angle formed by the spine relative to the knee with the pelvis as the axis of rotation:

[0152] The absolute values ​​of the angle differences between the first frame of the actual sit-up and the first frame of the standard sit-up in the XOY, XOZ, and YOZ planes are -147.0535 - (-151.5323), 159.4893 - 154.4737, and 0.7626 - (-3.4868), respectively, with results of 4.4788, 5.0156, and 4.2494.

[0153] The absolute values ​​of the angle differences between the actual sit-up frame 28 and the standard sit-up frame 26 in the XOY, XOZ and YOZ planes are -161.7457-(-164.3129), 233.7191-197.8144, and 56.1036-55.5697, respectively, with results of 2.5672, 35.9047, and 0.5339.

[0154] The absolute values ​​of the angle differences between the actual sit-up frame 53 and the standard sit-up frame 48 in the XOY, XOZ and YOZ planes are -149.5006 - (-154.7710), 164.5247 - 157.6072, and 2.6781 - (-3.8555), respectively, with results of 5.2704, 6.9175, and 6.5336.

[0155] The angles of the ankle, knee, and hip in the YOZ plane with the knee as the axis:

[0156] The absolute value of the angle difference between the first frame of the actual sit-up and the first frame of the standard sit-up in the YOZ plane is 185.9944-184.5603, resulting in 1.4341. The absolute value of the angle difference between the 28th frame of the actual sit-up and the 26th frame of the standard sit-up in the YOZ plane is 192.0269-189.0480, resulting in 2.9789. The absolute value of the angle difference between the 53rd frame of the actual sit-up and the 48th frame of the standard sit-up in the YOZ plane is 190.8554-182.0287, resulting in 8.8267.

[0157] Finally, the absolute values ​​of all these angle differences are added together: 1.0683, 0.9472, 1.9729, 0.3983, 0.1353, 1.2278, 4.4788, 5.0156, 4.2494, 2.5672, 35.9047, 0.5339, 5.2704, 6.9175, 6.5336, 1.4341, 2.9789, and 8.8267. The final result is 90.4606. When assessing the standard of sit-ups, this final value can be used for comparison; the smaller the value, the more standard the sit-ups.

Claims

1. A motion evaluation method based on keyframes and non-keyframes, characterized in that: Includes the following steps: Step 1: Export the 3D coordinates of each frame of the 32 joints of the human body acquired by the visual sensor; Step 2, select the key joints; Step 3: Correct the abnormal three-dimensional coordinates of the feature joints; Because visual sensors inevitably produce some coordinate anomalies when acquiring the 3D coordinates of skeletal joints, it is necessary to correct these anomalies. The correction method is as follows: if there are normal data on both sides of the anomaly, the average of the two closest normal data is taken during correction. If several anomalies occur consecutively, the first anomaly is corrected by taking the average of the two closest normal data on both sides. The second anomaly is corrected by taking the average of the corrected value of the first anomaly and the next closest normal data. The third anomaly is corrected by taking the average of the corrected value of the second anomaly and the next closest normal data, and so on, until all consecutive anomalies are corrected. Step 4: Calculate the angles between feature joints in each frame on the XOY, XOZ, and YOZ planes, and plot the change curves. Calculate the angles formed by the feature joints in each frame in the three planes XOY, XOZ, and YOZ. The feature joints refer to the feature joints that can represent the associated feature joints of a certain part and the corresponding angles formed by them. The angles formed between feature joints in each frame across the XOY, XOZ, and YOZ planes are calculated. To calculate the angles between several joints, their rotation axes need to be found. Then, the 3D coordinates with the rotation axis as the origin are obtained for each joint. Based on these new 3D coordinates, the angles formed between the joints and the corresponding coordinate axes in the three planes are calculated. The corresponding coordinate axes are the positive X-axis in the XOY plane, the positive X-axis in the XOZ plane, and the positive Y-axis in the YOZ plane. Finally, these angles are subtracted to obtain the included angles between the feature joints. The arctangent function is used to calculate the angle changes in the XOY, XOZ, and YOZ planes. In the XOY plane, using the formula Calculate the change in angle; In the XOZ plane, using the formula Calculate the change in angle; In the YOZ plane, using the formula Calculate the change in angle; use When calculating the angle, it is necessary to consider the specific quadrant in which its (A,B) coordinates lie. That is, when A>0, the angle can be calculated based on... The actual angle is obtained directly. When A < 0, the actual angle is... ; Step 5: If the curve consists of multiple actions, it needs to be divided into curves for individual actions. Step 6: Compare the curves of the actual movement with those of the standard movement, calculate the absolute value of the angle difference, and obtain the final evaluation result.

2. The motion evaluation method based on keyframes and non-keyframes according to claim 1, characterized in that: In step 2, when selecting feature joints, based on the characteristics of the movement itself, joints that can represent the characteristics and key points of the movement and can be easily evaluated are selected from among the 32 skeletal joints.

3. The motion evaluation method based on keyframes and non-keyframes according to claim 1, characterized in that: In step 5, based on the characteristics of the action itself, the angle change curve composed of multiple actions is segmented to separate the angle change curves of each action into the angle change curve of a single action.

4. The motion evaluation method based on keyframes and non-keyframes according to claim 1, characterized in that: In step 6, based on the characteristics of the action itself and the characteristics of the curve changes, the location of the keyframe can be determined relatively accurately; the angle change curves of the standard action feature joints projected onto the XOY, XOZ, and YOZ planes are compared with the angle change curves corresponding to the actual action. The specific steps are as follows: 6.1 Determine whether there is a curve in the standard motion angle change curve that can be regarded as a horizontal straight line; If there is a curve in the standard motion angle change curve that can be regarded as a horizontal straight line, find the curve in the actual motion angle change curve that has the same feature joint point and plane. First, determine whether the actual motion angle change curve can also be regarded as a horizontal straight line. If not, directly determine that the motion is unqualified and do not need to perform any further judgments. If yes, directly calculate the absolute value of the angle difference between the angle change curve corresponding to the standard motion at the keyframe. If there is no curve in the standard motion angle change curve that can be regarded as a horizontal straight line, proceed directly to the next step; 6.2 For the standard motion angle change curve, it cannot be regarded as a horizontal straight line. First, compare the angle change trend of the non-keyframe part of the actual motion angle change curve. If its angle change trend is roughly similar to the angle change curve corresponding to the standard motion, then it is only necessary to compare the keyframe part and calculate the absolute value of the angle difference between the keyframe and the standard motion. If the angle change trend is inconsistent with the angle change curve trend corresponding to the standard action, it is directly judged as an unqualified action, and there is no need to compare the key frame part. 6.3 Calculate the sum of the absolute values ​​of all angle differences; the smaller the sum, the more standard the movement. By combining the angle changes of non-keyframe parts, the angle change trends of non-keyframe parts are first compared, and then the keyframe parts are compared. When the non-keyframe parts meet the standards, the results are completely dominated by the keyframe parts. If the non-keyframe parts do not meet the standards, the evaluation results are completely determined by the non-keyframe parts.

5. A system used in the motion evaluation method based on keyframes and non-keyframes as described in any one of claims 1-4, characterized in that: It includes a data import module, a feature joint selection module, a data processing module, an angle calculation and curve drawing module, a curve processing module, and a motion evaluation module; The data import module can import the three-dimensional coordinates of each frame of the 32 skeletal joints of the human body obtained by the visual sensor into the computer. The feature joint selection module, while ensuring the accuracy of motion evaluation, selects some joints that can represent the motion from 32 skeletal joints based on the motion characteristics, referred to as feature joints. In order to prevent individual coordinate data anomalies from affecting the final evaluation result when the visual sensor collects the three-dimensional coordinates of human skeletal joints, the data processing module needs to correct the abnormal three-dimensional coordinates of the feature joints. The angle calculation and curve drawing module calculates the angles formed between feature joints in each frame on the three planes XOY, XOZ, and YOZ, and draws the angle change curves. If the obtained angle change curve is a curve of an action performed many times consecutively or a curve of multiple actions performed consecutively, the curve processing module needs to divide the curve into angle change curves of a single action; if the obtained curve is an angle change curve of a single action, this module is skipped directly. The motion evaluation module compares the trends of the actual motion angle change curve with the standard motion angle change curve at non-keyframes; at keyframes, it compares the angles of the actual motion with the standard motion; and combines these two parts to obtain the final evaluation result.

Images