Standardized pull-up evaluation method based on motion detection and multimodal learning

Abstract

The present invention discloses a standardized pull-up evaluation method based on action detection and multimodal learning, which mainly solves the problem of inaccurate pull-up counting and standard evaluation in existing methods. Its implementation scheme is: introducing face recognition for identity recognition, and controlling video acquisition through voice keywords; using a single-cycle waveform sequence segmentation method to perform single action detection on a complete pull-up action video, and extracting a single action video clip; setting action judgment parameters according to the assessment standard; performing human skeleton point estimation on the video clip, generating a single action evaluation vector, and judging the standard of a single pull-up action; performing multimodal learning of video, skeleton data and audio based on real-time judgment results; counting the number of pull-up actions that meet the standard, and generating an evaluation report to guide the tested person to view the standard example for action adjustment. The present invention has high detection accuracy and strong anti-interference ability, and can be used for daily physical training and physical examination.

Description

Technical Field

[0001] The invention belongs to the field of pattern recognition and machine learning, and mainly relates to a standardized evaluation method, which can be used for daily sports training and physical examination. Background Art

[0002] Pull-ups refer to hanging exercises that rely on one's own strength to overcome one's own weight and work upward. They are an important assessment item for men. Pull-ups mainly test the development level of upper limb muscle strength, as well as arm strength and waist and abdominal strength. They are a multi-joint compound movement exercise and one of the important reference standards and items for measuring male physique. The "National Student Physical Fitness Standards" require that pull-ups must be included in the physical examination of junior high school, high school and college boys, and stipulate specific movement specifications; the "Military Physical Training Outline" also stipulates that pull-ups are one of the physical training subjects that must be taken by officers and soldiers in the army. At the same time, because the venues and equipment required for pull-ups are not complicated and the exercise effect is good, pull-ups have become a fitness exercise for more and more people. The state has clearly stipulated the assessment standards for pull-ups. Standard pull-ups should meet the following three conditions: 1) The chin should be higher than the bar surface; 2) The body cannot be shaken or swung; 3) Both elbows should be straight when hanging. However, there is a lack of unified criteria for judging pull-up movements during assessments in various schools or the military. At the same time, incorrect movements in daily training can cause varying degrees of damage to the musculoskeletal tissue, which will affect physical fitness improvements in the long run and may even cause serious consequences.

[0003] In the process of pull-up training or assessment, the counting and standardization of the movements are very important. In the prior art, there are usually three ways to judge the number and standardization of pull-ups: a manual assessment method, a method based on auxiliary sensors, and a method based on computer vision.

[0004] The manual assessment method is for professional physical education teachers to count and standardize the assessment through visual observation and based on their own experience. This method requires one-to-one recording, which is labor-intensive and highly subjective. It is easy to cause fatigue to the supervisor, which may lead to inconsistent judgment standards and large counting errors. It is easy to make mistakes and cause unfair results. At the same time, since the data of the tested persons are recorded on paper and pen, there is a risk of missing data, and when there is a dispute over the judgment, it cannot be traced. In addition, cheating such as taking the exam for someone else may also occur.

[0005] The auxiliary sensor-based method generally installs infrared sensors, pressure sensors, and ultrasonic sensors around the horizontal bar, or asks the tested person to wear the corresponding sensors, analyzes the sensor information, and then completes the counting and judgment of the pull-up movements. This method has low equipment cost and unified evaluation standards, but the equipment is complex and the operation is not convenient. Wearing sensors will affect performance or cause safety accidents. In addition, this type of method still requires a person to guide and supervise, which is still manpower-consuming.

[0006] With the development of modern computer technology and artificial intelligence technology, computer vision-based methods have been widely used. This type of method first collects information such as videos and images of the pull-up process through non-contact data collection, and then designs intelligent algorithms to intelligently process and analyze the image information, thereby realizing the counting and evaluation of pull-ups. Among them, some technicians have proposed a pull-up counting method combined with depth images. The position of the horizontal bar and the jaw of the face is determined based on depth information and image segmentation to determine whether the jaw is higher than the bar surface, and whether the arm is straightened by the difference between the shoulder joint and the maximum height of the horizontal bar. However, this method lacks the judgment of whether the body is oscillating or swinging, which leads to the fact that such standardized evaluation is not comprehensive. In addition, the acquisition of depth images depends on the special depth image sensor Kinect, which is expensive and requires high computer performance. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the prior art and propose a pull-up intelligent counting and standardized evaluation method based on motion detection and multimodal learning, so as to avoid manpower consumption and complex sensor installation and improve the accuracy of standardized evaluation.

[0008] To achieve the above object, the technical solution of the present invention includes the following:

[0009] (1) Before the test begins, the face recognition algorithm is used to identify the person being tested to authenticate their identity. The system records the identity information of the person being tested. After the person being tested is ready to do pull-ups, the start and end of pull-up data collection is controlled according to the set voice keywords;

[0010] (2) According to the law that the body rises or falls with the change of the pull-up action, and the relatively stable characteristics of the key points of the eyes during the rise and fall process, the single-cycle waveform sequence segmentation method is used to detect the complete pull-up action video, obtain the start and end times of multiple single pull-up actions, and extract multiple video clips of single pull-up actions;

[0011] (3) According to the three major assessment criteria for pull-ups, set the judgment parameters for a single pull-up video clip:

[0012] (3a) According to the requirement in assessment standard 1 that the mandible of the person being tested should be higher than the bar surface, a determination parameter α is set to represent the position of the mandible on the bar surface, where α=1 means that the mandible is higher than the bar surface, and α=0 means that the mandible is lower than the bar surface;

[0013] (3b) According to the requirement in Assessment Standard 2 that the body of the person being tested cannot use the aid of waves or swings, a determination parameter β is set to represent the state of the body using the aid of waves or swings, β=1 represents that the body does not use the aid of waves or swings, and β=0 represents that the body uses the aid of waves or swings;

[0014] (3c) According to the requirement in assessment standard 3 that the elbow joints of the tested person should be straightened when in the hanging state, the judgment parameter γ is set to represent the state of the elbow joints in the hanging state, γ=1 means that the elbow joints are straightened when in the hanging state, and γ=0 means that the elbow joints are not straightened when in the hanging state;

[0015] (4) Perform human skeleton point estimation on the video clip of a single pull-up action and generate an evaluation vector [α, β, γ] of the single pull-up action:

[0016] (4a) Extract facial key points from the pull-up video clip, use the lip points of the facial key points to approximate the position of the jaw point, and determine the Y coordinate y of the center of the horizontal bar during the pull-up process in real time b The Y coordinate y of the jaw point of the tested person m The relative position of m ≥y b , then α=1, otherwise α=0;

[0017] (4b) Three conditions are set to determine whether the body is supported by waves or swings:

[0018] (4b1) Assume that the upper body tilt angle threshold is Th1, calculate the standard deviation σ1 of the upper body tilt angle, if σ1 ≥ Th1, the upper body swings too much, β = 0;

[0019] (4b2) Let the lower body tilt angle threshold be Th2, calculate the standard deviation σ2 of the lower body tilt angle, if σ2 ≥ Th2, the lower body swings too much, β = 0;

[0020] (4b3) Let the knee bending angle threshold be Th3, calculate the standard deviation of the knee bending angle σ3, if σ3 ≥ Th3, there is a kicking phenomenon, β = 0;

[0021] If there is no excessive swing of the upper and lower body and kicking of the legs, β = 1;

[0022] (4c) Setting the joint angle threshold Th θ, a single pull-up action is divided into an ascending process and a descending process, human bone data is extracted during the descending process, and the angle of the elbow joint of the tested person at time t during the descending process of the pull-up action is calculated in real time like Then γ=1, otherwise γ=0;

[0023] (4d) For a single pull-up video clip in which the chin of the tested person is higher than the bar surface during the pull-up process, the body is not swung or swung, and the elbows are straightened in the hanging state, that is, α=β=γ=1, and its evaluation vector [α, β, γ]=[1,1,1] is obtained; for a single pull-up video clip in which the chin of the tested person is lower than the bar surface during the pull-up process, the body is swung or swung, and the elbows are not straightened in the hanging state, that is, α=β=γ=0, and its evaluation vector [α, β, γ]=[0,0,0] is obtained;

[0024] (5) Based on the evaluation vector [α, β, γ] of each single pull-up action generated during the entire pull-up process, determine whether each single pull-up action meets the standard:

[0025] If the evaluation vector [α, β, γ] of a single pull-up video clip = [1, 1, 1], it means that the pull-up action meets the standard;

[0026] If the evaluation vector [α, β, γ] ≠ [1, 1, 1], it means that the pull-up action does not meet the standard;

[0027] (6) Control the audio output according to the standardized judgment results of the pull-up video action, and realize the multimodal learning between video, skeleton data and audio based on the audio output results. For example: when it is detected that the detection of three consecutive pull-ups does not meet the standard, control the audio output to send voice keywords such as "please re-record" and then continue the multimodal action detection learning of video and skeleton;

[0028] (7) Count the number of single pull-up video clips that meet the standard movements, and obtain the number of standard movements of the test subject during the entire pull-up training process. Generate an evaluation report based on the number of standard movements, provide examples of pull-up videos with non-standard movements of the test subject and the reasons for the non-standard movements, and provide intelligent improvement suggestions for the entire pull-up process;

[0029] (8) Save the pull-up standard counting results, evaluation vectors and videos of all tested persons and upload them to the database.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] 1. Able to adapt to a variety of scenarios:

[0032] The present invention introduces face recognition technology. Before the test starts, the face recognition algorithm is used to perform face recognition on the tested person, and the face features of the tested person are compared with the face features in the database. The system function is enabled only when the comparison is successful, and the face comparison is continuously monitored during the pull-up process to see whether it is the same person, so that cheating can be better prevented. At the same time, the voice keyword control system function is introduced, so the start and end of the collection can be controlled according to the set voice keyword. In addition, because the pull-up is evaluated by introducing a threshold, different thresholds can be set in different scenes to realize the evaluation of different difficulty levels. The present invention can be used for daily training and exercise, as well as for physical examination and assessment, to realize unmanned supervision.

[0033] 2. Ability to analyze each action in detail:

[0034] The single-cycle waveform sequence segmentation method proposed in the present invention is used to detect a complete pull-up action video, and can segment the complete pull-up action video into multiple single action segments, so as to finely analyze every action detail of the tested person.

[0035] 3. Standard action count:

[0036] The present invention refers to the three major assessment standards for pull-ups. By detecting a single pull-up action, it is determined whether the lower jaw of the tested person is higher than the bar surface, whether the elbows are straight when hanging, whether the upper and lower body are swinging, and whether the legs are kicked. The standardization of the pull-up action can be comprehensively evaluated to obtain a standard counting result.

[0037] 4. Multimodal learning and interaction:

[0038] The present invention controls audio output according to the standardized judgment result of the pull-up video action, controls multimodal action detection of video and skeleton data based on the audio output, and realizes multimodal learning and interaction among video, skeleton data and audio.

[0039] 5. Humanized evaluation report:

[0040] The present invention generates an evaluation report at the end of the test, which provides video examples of pull-ups with non-standard movements of the tested person and the reasons for the non-standard movements, and simultaneously provides intelligent improvement suggestions for the entire pull-up process, guiding the tested person to view standard examples and make movement adjustments, thereby ensuring that the tested person can promptly discover and correct his or her own movement errors during training.

[0041] 6. Data traceability:

[0042] The present invention saves the pull-up standard counting results, evaluation vectors and videos of the tested person in a computer and can upload them to a database, so that the tested person can grasp his or her training situation at each stage and can check it when there is a dispute in the assessment.

[0043] 7. Non-contact testing:

[0044] The present invention detects the pull-up action through computer vision, has high detection accuracy, strong anti-interference ability, high safety, low cost of video shooting equipment and simple installation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of the implementation process of the present invention;

[0046] Figure 2 is a schematic diagram of the natural stretching state in the pull-up evaluation of the present invention;

[0047] Figure 3 is a schematic diagram of body swing inclination in the pull-up evaluation of the present invention;

[0048] Figure 4 It is a schematic diagram of the leg-pushing state in the pull-up evaluation of the present invention;

[0049] Figure 5 Schematic diagram of the test scenario of the present invention. DETAILED DESCRIPTION

[0050] Under the background of rapid development of artificial intelligence technology and high attention paid to human health, standardized pull-up counting methods play a more important role in sports testing and assessment, and there is an urgent need for an efficient, accurate and intelligent standardized pull-up counting method. At present, manual supervision requires one-to-one recording, which is labor-intensive and highly subjective. The auxiliary sensor equipment is complex to install, has poor operation convenience and general accuracy. Computer vision-based methods often do not consider whether the body is judged by vibration or swing, and do not analyze the pull-up action or make suggestions for improvement. In view of the current situation, the present invention, after exploration and experimentation, proposes a standardized pull-up evaluation method based on motion detection and multimodal learning, which is used for intelligent counting and standardized evaluation of pull-ups in a wide range of applicable scenarios.

[0051] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0052] See also Figure 1 , the implementation steps of this example are as follows:

[0053] Step 1: Identity recognition and video acquisition.

[0054] Before the test begins, fix the camera 2 to 3 meters in front of the pull-up bar and connect the camera to the computer;

[0055] After the camera is installed, turn on the camera in the computer, use the face recognition algorithm to perform face recognition on the image frames collected by the camera, obtain the face features of the person being tested, and compare the face features with the face features in the database. If the face features match, the system records the identity information of the person being tested, the system function is officially turned on, and the system voice prompts the person being tested to start the pull-up test;

[0056] After the person being tested is ready for pull-ups, the start and end of pull-up data collection are controlled according to the set voice keywords.

[0057] Step 2: Extract the video clip of a single pull-up action.

[0058] In a training or assessment, the pull-up test video of the test subject usually includes multiple pull-ups. In order to accurately evaluate each action, before evaluating the standardization of the pull-up action, it is necessary to extract each single pull-up action video clip for subsequent analysis.

[0059] In order to solve the problem that multiple actions cannot be analyzed when detecting a single action at the same time, the present invention uses a single-cycle waveform sequence segmentation method to detect the complete pull-up action video according to the law that the body rises or falls with the change of the action during the pull-up action, and the relatively stable characteristics of the key points of the eyes during the rise and fall process, and obtains the start and end times of multiple single pull-up actions, and extracts multiple video clips of single pull-up actions. The specific implementation is as follows:

[0060] (2.1) The pull-up video is processed using the MediaPipe 3D skeleton estimation algorithm to obtain the Y coordinate sequence Y0 of the eye key points during the entire pull-up process.

[0061] (2.2) Smooth and denoise the Y0 sequence:

[0062] Since the MediaPipe 3D skeleton estimation algorithm sometimes has skeleton confusion problems when extracting skeleton key points, the eye key point Y0 contains a small number of error points, so denoising is required before analysis;

[0063] In this example, the Savitzky-Golay filter is used to smooth and denoise the Y0 sequence to obtain the denoised Y coordinate sequence Y1; the Savitzky-Golay filter filtering method can filter out noise while ensuring that the shape and width of the signal remain unchanged, which is conducive to reducing the counting error of the subsequent evaluation algorithm.

[0064] (2.3) Find the subsequence Y of the trough and peak points of the Y1 sequence tc , and remove its error points:

[0065] Since the waveform of the denoised sequence Y1 is similar to that of a cosine wave, there are obvious troughs and peaks in the denoised sequence Y1. According to the rule that the key points of the eyes rise or fall with the change of the movement during the pull-up action, the trough of the sequence Y1 corresponds to the frame with the highest point of the pull-up, and the peak corresponds to the end frame of the pull-up. Therefore, it is necessary to find the trough and peak set subsequence Y of the Y1 sequence. tc ;

[0066] Set the threshold of the difference between two points in the sequence to Th δ Since there are still a few noise points after denoising the Y1 sequence, we add limiting conditions to reduce the error, that is, to find Y tc The difference between each point and its previous point in the , remove the absolute value of the difference less than the set threshold Th δ The error points are obtained to obtain the subsequence Y of the trough points and peak points after removing the error points tc ';

[0067] (2.4) According to Y tc 'The size relationship of each point in the subsequence, Y tc 'Divided into trough point sequence Y t and the peak point sequence Y c ; Set the frame interval of each single action in the entire pull-up process to 1, and use the next frame after the end frame of each single action as the start frame of the next single action, that is, Y c With Y c+1 They correspond to the end frame of each single action and the start frame of the next single action respectively;

[0068] (2.5) For the peak point sequence Y c Segmentation is performed, and the pull-up action video is cropped according to the segmentation result to obtain each single pull-up action video clip.

[0069] Compared with the existing pull-up evaluation method, this example introduces a method with a single pull-up action as the basic detection unit. In order to solve the problem that multiple actions cannot be analyzed simultaneously, a single pull-up action video clip is extracted. Analyzing the single pull-up action can better analyze each action of the tested person.

[0070] Step 3, setting the judgment parameters of the single pull-up video clip action.

[0071] According to the national standards for pull-ups, a standard pull-up should meet the following three conditions: 1) the chin should be above the bar; 2) the body should not swing or sway; 3) both elbows should be straight when hanging. This example sets the judgment parameters for a single pull-up video clip action based on this regulation:

[0072] (3.1) According to the requirement in assessment standard 1 that the mandible of the person being tested should be higher than the bar surface, the judgment parameter α is set to represent the position of the mandible on the bar surface, α=1 means that the mandible is higher than the bar surface, and α=0 means that the mandible is lower than the bar surface;

[0073] (3.2) According to the requirement in assessment standard 2 that the body of the person being tested cannot use the aid of waves or swings, the judgment parameter β is set to represent the state of the body using the aid of waves or swings, β=1 means that the body does not use the aid of waves or swings, and β=0 means that the body uses the aid of waves or swings;

[0074] (3.3) According to the requirement in assessment standard 3 that the elbow joints of the tested person need to be straightened when in a hanging state, the judgment parameter γ is set to represent the state of the elbow joints in the hanging state, γ=1 means that the elbow joints are straightened when in a hanging state, and γ=0 means that the elbow joints are not straightened when in a hanging state.

[0075] Step 4, intelligently evaluate the single pull-up video clip of the tested person to generate definite values ​​of three judgment parameters.

[0076] (4.1) The MediaPipe 3D skeleton estimation algorithm is used to extract facial key points from a single pull-up video clip, and the lip points of the facial key points are used to approximate the position of the jaw point, and the Y coordinate y of the center of the horizontal bar during the pull-up process is determined in real time. b The Y coordinate y of the jaw point of the tested person m The relative position of m ≥y b , then α=1, otherwise α=0;

[0077] In this example, the lip key point is used to replace the jaw key point for judgment. Since the lips are above the jaw key point, the empirical value of the difference between the Y coordinates of the lip and jaw key points can be obtained. Therefore, in the actual test, the Y coordinate of the horizontal bar is obtained and then the empirical value is subtracted. At this time, if the lip point is higher than the bar surface, it can be approximately indicated that the jaw is higher than the bar surface.

[0078] (4.2) Three judgment conditions are set for whether the body is supported by waves or swings, and the judgment parameter value β is determined:

[0079] (4.2.1) Assume that the upper body tilt angle threshold is Th1, and calculate the standard deviation σ1 of the upper body tilt angle:

[0080] (4.2.1a) Use the target detection algorithm to identify the horizontal bar in the video and mark its position in the video. Select the center point of the horizontal bar as the coordinate of the horizontal bar, and set the Y coordinate of the horizontal bar to be y b ;

[0081] (4.2.1b) The MediaPipe 3D skeleton estimation algorithm is used to extract the key points of the human shoulder, waist and thumb in each frame of the video. The 3D coordinates of the shoulder at time t are set to The three-dimensional coordinates of the waist are The three-dimensional coordinates of the thumb are

[0082] (4.2.1c) Let the start flag f s The initial value is 0, according to the start flag f s The value of and the relationship between the thumb key point and the Y coordinate of the horizontal bar are used to determine the state of the pull-up:

[0083] like And f s =0, it indicates the initial state of pull-up;

[0084] like And f s =1, it indicates the process state of pull-up;

[0085] (4.2.1d) See Figure 2 , at the initial state of the pull-up, the three-dimensional coordinates of the shoulder are The three-dimensional coordinates of the waist are Assume the initial length of the upper body is L us , at this time the initial length of the upper body is L us It is equal to the absolute value of the difference between the Y coordinates of the shoulder and waist, that is, And will start the mark f s Set to 1;

[0086] (4.2.1e) See Figure 3 , in the process of pull-up, let the three-dimensional coordinates of the shoulder at a certain time d be The three-dimensional coordinates of the waist are Assume the current length of the upper body is Current length of upper body is the absolute value of the difference between the Y coordinates of the shoulder and waist, that is

[0087] (4.2.1f) The initial length of the upper body is L us , Current length of upper body Connect the ends of these two to form a right triangle, and set the upper body deviation angle at time d to According to the characteristics of the right triangle and the arccosine function, calculate the deviation angle of the upper body of the tested person

[0088]

[0089] (4.2.1g) Calculate the upper body deviation angle at each moment in the pull-up process Get the upper body deviation angle sequence Y u ;

[0090] (4.2.1h) Upper body deviation angle sequence Y u Calculate the standard deviation and obtain the standard deviation of the upper body tilt angle σ1;

[0091] (4.2.1i) Compare the standard deviation σ1 of the upper body tilt angle with the upper body tilt angle threshold Th1:

[0092] If σ1 ≥ Th1, it means that the upper body swings too much, β = 0;

[0093] If σ1<Th1, it means that the upper body does not swing much, β=1.

[0094] (4.2.2) Assume the lower body tilt angle threshold Th2, and calculate the standard deviation σ2 of the lower body tilt angle:

[0095] (4.2.2a) Use the target detection algorithm to identify the horizontal bar in the video and mark its position in the video. Select the center point of the horizontal bar as the coordinate of the horizontal bar. Set the Y coordinate of the horizontal bar as y b ;

[0096] (4.2.2b) The MediaPipe 3D skeleton estimation algorithm is used to extract the key points of the waist, ankles and thumbs of each frame of the video. Let the 3D coordinates of the waist at time t be The three-dimensional coordinates of the ankle are The three-dimensional coordinates of the thumb are

[0097] (4.2.2c) Let the start flag f s The initial value is 0, according to the start flag f s The value of and the relationship between the thumb key point and the Y coordinate of the horizontal bar are used to determine the state of the pull-up:

[0098] like And f s =0, it indicates the initial state of pull-up;

[0099] like And f s =1, it indicates the process state of pull-up;

[0100] (4.2.2d) At the initial state of the pull-up, the three-dimensional coordinates of the waist are The three-dimensional coordinates of the ankle are Assume the initial length of the lower body is L ds , at this time the initial length of the lower body is L ds is equal to the absolute value of the difference between the waist and ankle Y coordinates, that is, And will start the mark f s Set to 1;

[0101] (4.2.2e) In the process of pull-up, let the three-dimensional coordinates of the waist at a certain time d be The three-dimensional coordinates of the ankle are Let the current length of the lower body be Current length of lower body is the absolute value of the difference between the Y coordinates of the shoulder and waist, that is

[0102] (4.2.2f) The initial length of the lower body is L ds , Current length of lower body Connect the ends of these two to form a right triangle, and set the deviation angle of the lower body at time d to According to the characteristics of the right triangle and the arccosine function, calculate the deviation angle of the lower body of the person being tested

[0103]

[0104] (4.2.2g) Calculate the deviation angle of the lower body at each moment in the pull-up process Get the lower body deviation angle sequence Y l ;

[0105] (4.2.2h) for the lower body deviation angle sequence Y l Calculate the standard deviation and obtain the standard deviation of the inclination angle of the lower body σ2;

[0106] (4.2.2i) Compare the standard deviation σ2 of the lower body tilt angle with the lower body tilt angle threshold Th2:

[0107] If σ2 ≥ Th2, it means that the lower body swings too much, β = 0;

[0108] If σ2<Th2, it means that the lower body does not swing much, β=1.

[0109] (4.2.3) See Figure 4 , set the knee bending angle threshold Th3, calculate the standard deviation σ3 of the knee bending angle:

[0110] (4.2.3a) During the pull-up process, the MediaPipe 3D skeleton estimation algorithm is used in real time to extract the waist, knee and ankle key points of each frame of the video. Let the 3D coordinates of the waist at time t be The three-dimensional coordinates of the knee are The three-dimensional coordinates of the ankle are

[0111] (4.2.3b) Connect the key points of the waist, knees, and ankles to form a triangle, calculate the three sides of the triangle, and assume that the distance from the waist to the knee at time t is The distance from waist to ankle is The distance from knee to ankle is The calculation formula is as follows:

[0112]

[0113]

[0114]

[0115] (4.2.3c) Set the knee bending angle at time t to According to the side length of the triangle formed by the waist, knee and ankle key points and the cosine theorem of the triangle, the knee bending angle is calculated. The cosine of :

[0116]

[0117] (4.2.3d) Calculate the knee bending angle using the arccosine function

[0118]

[0119] (4.2.3e) Calculate the knee bending angle at each moment during the pull-up movement Get the knee bending angle sequence Y k ;

[0120] (4.2.3f) for the knee bending angle sequence Y k Calculate the standard deviation and obtain the standard deviation of the knee bending angle σ3;

[0121] (4.2.3g) Compare the standard deviation σ3 of the knee angle with the knee angle threshold Th3:

[0122] If σ3 ≥ Th3, it means that there is a kicking phenomenon, β = 0;

[0123] If σ3<Th3, it means that there is no kicking phenomenon, and β=1.

[0124] (4.3) Set the joint angle threshold Th θ , according to whether the lower jaw is above the bar surface during this action, a single pull-up action is divided into an ascending process and a descending process. If the lower jaw is above the bar surface, it means that the pull-up is in the descending process;

[0125] (4.4) Real-time calculation of the angle of the elbow joint of the person being tested at time t during the descent of the pull-up action Determine the decision parameter value γ:

[0126] (4.4.1) Use the MediaPipe 3D skeleton estimation algorithm to extract the key points of the shoulder, elbow, and wrist. Suppose the 3D coordinates of the shoulder at time t are The three-dimensional coordinates of the elbow are The three-dimensional coordinates of the wrist key points are

[0127] (4.4.2) Connect the key points of the shoulder, elbow, and wrist to form a triangle, calculate the three sides of the triangle, and assume that the distance from the shoulder to the elbow at time t is The distance from shoulder to wrist is The distance from elbow to wrist is The calculation formula is as follows:

[0128]

[0129]

[0130]

[0131] (4.4.3) Assume that the elbow bending angle is According to the side length of the triangle formed by the shoulder, elbow and wrist key points and the cosine theorem of the triangle, calculate the elbow bending angle The cosine of :

[0132]

[0133] (4.4.4) Calculate the elbow bending angle using the arccosine function

[0134]

[0135] (4.4.5) Bend your elbows and joint angle threshold Th θ For comparison:

[0136] like This means that both elbows have been straightened during the descent, and γ = 1;

[0137] like This means that both elbows were not straightened during the descent, and γ = 0.

[0138] Step 5: Generate an evaluation vector [α, β, γ] to judge the standardization of the single pull-up action video clip.

[0139] (5.1) According to the judgment parameter value determined in step 4, different evaluation vectors are generated:

[0140] For a single pull-up video clip in which the subject's chin is above the bar, the body is not oscillating or swinging, and both elbows are straight in a suspended state, that is, α=β=γ=1, an evaluation vector [α, β, γ]=[1,1,1] is generated;

[0141] For a single pull-up video clip in which the chin of the person being tested is higher than the bar surface, the body is not swung or swinging, and the elbows are not straightened when the person is hanging, that is, α=β=1, γ=0, an evaluation vector [α, β, γ]=[1,1,0] is generated;

[0142] For a single pull-up video clip in which the chin of the person being tested is above the bar surface and the body is suspended with both elbows straight, i.e., α = γ = 1, β = 0, an evaluation vector [α, β, γ] = [1, 0, 1] is generated;

[0143] For a single pull-up video clip in which the chin of the person being tested is higher than the bar surface during the pull-up process, the body is oscillating or swinging, and the elbows are not straightened when the body is in a suspended state, that is, α=1, β=γ=0, an evaluation vector [α,β,γ]=[1,0,0] is generated;

[0144] For a single pull-up video clip in which the subject's chin is below the bar surface and both elbows are straight while the body is suspended without any vibration or swing, i.e., α = 0, β = γ = 1, an evaluation vector [α, β, γ] = [0, 1, 1] is generated;

[0145] For a single pull-up video clip in which the test subject's chin is below the bar surface during the pull-up process, the body is not swung or swinging, and the elbows are not straightened when in the hanging state, that is, α=γ=0, β=1, an evaluation vector [α, β, γ]=[0,1,0] is generated;

[0146] For a single pull-up video clip in which the chin of the person being tested is below the bar surface and the body is suspended with both elbows straight, i.e., α = β = 0, γ = 1, an evaluation vector [α, β, γ] = [0, 0, 1] is generated;

[0147] For a single pull-up video clip in which the test subject's chin is below the bar surface and both elbows are not straightened while the body is oscillating or swinging and hanging, that is, α=β=γ=0, an evaluation vector [α,β,γ]=[0,0,0] is generated.

[0148] (5.2) Judge the standardization of each single pull-up video clip based on the evaluation vector:

[0149] If the evaluation vector [α, β, γ] generated by a single pull-up video clip = [1, 1, 1], it means that the pull-up action meets the standard, and the single pull-up video clip is recorded as meeting the action standard;

[0150] If the evaluation vector [α, β, γ]≠[1,1,1] generated by a single pull-up video clip, it means that the pull-up action does not meet the standard, and the single pull-up video clip is recorded as not meeting the action standard.

[0151] Step 6: Multimodal learning between video, skeleton data and audio.

[0152] The audio output is controlled according to the real-time judgment result of the pull-up video action standardization. When it is detected that the pull-up action detection fails to meet the standard for three consecutive times, the audio output is controlled to request re-recording and other voice keywords;

[0153] During re-recording, when voice keywords such as start recording uttered by the person being tested are detected, multimodal motion detection of video and skeleton is continued to achieve multimodal learning between video, skeleton data and audio.

[0154] Step 7: Count the number of standards and generate an evaluation report.

[0155] The number of single pull-up video clips that meet the action standards is counted to obtain the number of standard actions of the tested person during the entire pull-up training process;

[0156] An evaluation report is generated based on the standard number of pull-ups and evaluation vectors of each single pull-up of the test subject. The evaluation report provides video examples of pull-ups with non-standard movements of the test subject and the reasons for the non-standard movements. At the same time, intelligent improvement suggestions are given for the entire pull-up process, guiding the test subject to review the standard examples and make movement adjustments, so as to ensure that the test subject can promptly discover and correct his or her own movement errors during training.

[0157] Step 8: Save the data file and upload it to the database.

[0158] During the pull-up process, the pull-up standard counting results, evaluation vectors and video paths of the tested person are all stored in a list;

[0159] After the evaluation is completed, the pull-up standard counting results, evaluation vectors and video paths in the list are saved in Excel. The Pandas.DataFrame method is used to operate Excel, and the pull-up standard counting results, evaluation vectors and video paths of all tested persons are saved in Excel, and uploaded to the database using MySql technology, so that the tested persons can understand their training status at each stage and check when there is a dispute in the assessment.

[0160] The effect of the present invention can be further illustrated by the following test experiments:

[0161] 1. Test conditions:

[0162] Test scenarios such as Figure 5 As shown, it contains a pull-up bar;

[0163] The test equipment is a computer and a camera. The computer and the camera are connected and set up in front of the venue containing the pull-up bar, and it is best to set them up 2 to 3 meters in front.

[0164] The test data is pull-up videos of 51 test subjects collected at a university. The true values ​​of the standard number of pull-ups of all test subjects were marked by hospital clinicians.

[0165] The simulation test platform is the 11th Gen Intel(R)Core(TM)i7-1165G7 CPU 2.80GHz, the memory is 32GB, the test platform is the Windows10 operating system, and it is implemented in Python language.

[0166] Evaluation index: It is defined as the average value of the true value of the standard number of pull-ups completed by each tested person marked by the doctor in the test data and the sum of the errors of the standard number of pull-ups completed by each tested person obtained by the method of the present invention.

[0167] 2. Test content and results:

[0168] Under the above test conditions, the method of the present invention is used to process the pull-up video data of 51 tested persons, and the standard number of pull-ups and evaluation reports of all tested persons are obtained. The average value of the sum of the differences between the standard number of pull-ups of each tested person and the true value of the standard number of pull-ups marked by the doctor is calculated, and the standardized average error is 0.75. The error value is less than 1, indicating that the test result is not much different from the actual true value, and the standardized pull-up evaluation has high accuracy.

[0169] In summary, the standardized pull-up evaluation method based on motion detection and multimodal learning described in the present invention has accurate testing and high robustness.

Claims

1. A standardized pull-up evaluation method based on motion detection and multimodal learning, characterized in that: These include: (1) Before the test begins, the face recognition algorithm is used to identify the person being tested to authenticate their identity. The system records the identity information of the person being tested. After the person being tested is ready to do pull-ups, the start and end of pull-up data collection is controlled according to the set voice keywords; (2) According to the law that the body rises or falls with the change of the pull-up action, and the relatively stable characteristics of the key points of the eyes during the rise and fall process, the single-cycle waveform sequence segmentation method is used to detect the complete pull-up action video, obtain the start and end times of multiple single pull-up actions, and extract multiple video clips of single pull-up actions; (3) According to the three major assessment criteria for pull-ups, set the judgment parameters for a single pull-up video clip: (3a) According to the requirement in assessment standard 1 that the mandible of the person being tested should be higher than the bar surface, a determination parameter α is set to represent the position of the mandible on the bar surface, where α=1 means that the mandible is higher than the bar surface, and α=0 means that the mandible is lower than the bar surface; (3b) According to the requirement in Assessment Standard 2 that the body of the person being tested cannot use the aid of waves or swings, a determination parameter β is set to represent the state of the body using the aid of waves or swings, β=1 represents that the body does not use the aid of waves or swings, and β=0 represents that the body uses the aid of waves or swings; (3c) According to the requirement in assessment standard 3 that the elbow joints of the tested person should be straightened when in the hanging state, the judgment parameter γ is set to represent the state of the elbow joints in the hanging state, γ=1 means that the elbow joints are straightened when in the hanging state, and γ=0 means that the elbow joints are not straightened when in the hanging state; (4) Perform human skeleton point estimation on the video clip of a single pull-up action and generate an evaluation vector [α, β, γ] of the single pull-up action: (4a) Extract facial key points from the pull-up video clip, use the lip points of the facial key points to approximate the position of the jaw point, and determine the Y coordinate y of the center of the horizontal bar during the pull-up process in real time b The Y coordinate y of the jaw point of the tested person m The relative position of m ≥y b , then α=1, otherwise α=0; (4b) Three conditions are set to determine whether the body is supported by waves or swings: (4b1) Assume that the upper body tilt angle threshold is Th1, calculate the standard deviation σ1 of the upper body tilt angle, if σ1 ≥ Th1, the upper body swings too much, β = 0; (4b2) Let the lower body tilt angle threshold be Th2, calculate the standard deviation σ2 of the lower body tilt angle, if σ2 ≥ Th2, the lower body swings too much, β = 0; (4b3) Let the knee bending angle threshold be Th3, calculate the standard deviation of the knee bending angle σ3, if σ3 ≥ Th3, there is a kicking phenomenon, β = 0; If there is no excessive swing of the upper and lower body and kicking of the legs, β = 1; (4c) Setting the joint angle threshold Th θ , a single pull-up action is divided into an ascending process and a descending process, human bone data is extracted during the descending process, and the angle of the elbow joint of the tested person at time t during the descending process of the pull-up action is calculated in real time like Then γ=1, otherwise γ=0; (4d) For a single pull-up video clip in which the chin of the tested person is higher than the bar surface during the pull-up process, the body is not swung or swung, and the elbows are straightened in the hanging state, that is, α=β=γ=1, and its evaluation vector [α, β, γ]=[1,1,1] is obtained; for a single pull-up video clip in which the chin of the tested person is lower than the bar surface during the pull-up process, the body is swung or swung, and the elbows are not straightened in the hanging state, that is, α=β=γ=0, and its evaluation vector [α, β, γ]=[0,0,0] is obtained; (5) Based on the evaluation vector [α, β, γ] of each single pull-up action generated during the entire pull-up process, determine whether each single pull-up action meets the standard: If the evaluation vector [α, β, γ] of a single pull-up video clip = [1, 1, 1], it means that the pull-up action meets the standard; If the evaluation vector [α, β, γ] ≠ [1, 1, 1], it means that the pull-up action does not meet the standard; (6) Controlling audio output according to the standardized judgment results of the pull-up video movements, and realizing multimodal learning between video, bone data and audio based on the audio output results; (7) Count the number of single pull-up video clips that meet the standard movements, and obtain the number of standard movements of the test subject during the entire pull-up training process. Generate an evaluation report based on the number of standard movements, provide examples of pull-up videos with non-standard movements of the test subject and the reasons for the non-standard movements, and provide intelligent improvement suggestions for the entire pull-up process; (8) Save the pull-up standard counting results, evaluation vectors and videos of all tested persons and upload them to the database.

2. The method according to claim 1, characterized in that In (2), the single-cycle waveform sequence segmentation method is used to detect the complete pull-up action video, which is implemented as follows: (2a) Use the MediaPipe 3D skeleton estimation algorithm to obtain the sequence Y0 of the Y coordinates of the eye key points during the pull-up process; (2b) Use Savitzky-Golay filter to smooth and denoise the Y0 sequence to obtain the denoised Y coordinate sequence Y1; (2c) Find the subsequence Y of the trough and peak points of the Y1 sequence tc ; (2d) Find Y tc The difference between each point and its previous point in the , remove the absolute value of the difference less than the set threshold Th δ The error points are obtained to obtain the subsequence Y of the trough points and peak points after removing the error points tc '; (2e) Y tc 'Divided into trough point sequence Y t and the peak point sequence Y c , Y c Corresponding to the end frame of each single pull-up, the next frame of each single action end frame is used as the start frame of the next single action, that is, Y c With Y c+1 They correspond to the end frame of each single action and the start frame of the next single action respectively; (2f) For the peak point sequence Y c Segmentation is performed, and the pull-up action video is cropped according to the segmentation result to obtain each single pull-up action video clip.

3. The method according to claim 1, characterized in that The standard deviation σ1 of the upper body tilt angle is calculated in (4b1) as follows: (4b1.1) Use the target detection algorithm to identify the horizontal bar in the video and mark its position in the video. Select the center point of the horizontal bar as the coordinate of the horizontal bar. Set the Y coordinate of the horizontal bar to be y b ; (4b1.2) Use the MediaPipe 3D skeleton estimation algorithm to extract the key points of the human shoulder, waist, and thumb in each frame of the video. Suppose the 3D coordinates of the shoulder at time t are The three-dimensional coordinates of the waist are The three-dimensional coordinates of the thumb are (4b1.3) Let the start mark f s The initial value is 0, according to the start flag f s The value of and the relationship between the thumb key point and the Y coordinate of the horizontal bar are used to determine the state of the pull-up: like And f s =0, it indicates the initial state of pull-up; like And f s =1, it indicates the process state of pull-up; (4b1.4) At the initial state of the pull-up, the three-dimensional coordinates of the shoulder are The three-dimensional coordinates of the waist are Assume the initial length of the upper body is L us , at this time the initial length of the upper body is L us It is equal to the absolute value of the difference between the Y coordinates of the shoulder and waist, that is, And will start the mark f s Set to 1; (4b1.5) In the pull-up process, let the three-dimensional coordinates of the shoulder at a certain time d be The three-dimensional coordinates of the waist are Assume the current length of the upper body is Current length of upper body is the absolute value of the difference between the Y coordinates of the shoulder and waist, that is (4b1.6) The initial length of the upper body is L us , Current length of upper body Connect the ends of these two to form a right triangle, and set the upper body deviation angle at time d to According to the characteristics of the right triangle and the arccosine function, calculate the deviation angle of the upper body of the tested person (4b1.7) Calculate the upper body deviation angle at each moment in the pull-up process Get the upper body deviation angle sequence Y u ; (4b1.8) for the upper body deviation angle sequence Y u Find the standard deviation and obtain the standard deviation σ1 of the inclination angle of the upper body.

4. The method according to claim 1, characterized in that: The standard deviation σ2 of the inclination angle of the lower body is calculated in (4b2) as follows: (4b2.1) Use the target detection algorithm to identify the horizontal bar in the video and mark its position in the video. Select the center point of the horizontal bar as the coordinate of the horizontal bar. Set the Y coordinate of the horizontal bar to be y b ; (4b2.2) Use the MediaPipe 3D skeleton estimation algorithm to extract the key points of the waist, ankles, and thumbs of each frame of the video. Suppose the 3D coordinates of the waist at time t are The three-dimensional coordinates of the ankle are The three-dimensional coordinates of the thumb are (4b2.3) Let the start mark f s The initial value is 0, according to the start flag f s The value of and the relationship between the thumb key point and the Y coordinate of the horizontal bar are used to determine the state of the pull-up: like And f s =0, it indicates the initial state of pull-up; like And f s =1, it indicates the process state of pull-up; (4b2.4) At the initial state of the pull-up, the three-dimensional coordinates of the waist are The three-dimensional coordinates of the ankle are Assume the initial length of the lower body is L ds , at this time the initial length of the lower body is L ds is equal to the absolute value of the difference between the waist and ankle Y coordinates, that is, And will start the mark f s Set to 1; (4b2.5) During the pull-up process, let the three-dimensional coordinates of the waist at a certain time d be The three-dimensional coordinates of the ankle are Let the current length of the lower body be Current length of lower body is the absolute value of the difference between the Y coordinates of the shoulder and waist, that is (4b2.6) The initial length of the lower body is L ds , Current length of lower body Connect the ends of these two to form a right triangle, and set the deviation angle of the lower body at time d to According to the characteristics of the right triangle and the arccosine function, calculate the deviation angle of the lower body of the person being tested (4b2.7) Calculate the deviation angle of the lower body at each moment in the pull-up process Get the lower body deviation angle sequence Y l ; (4b2.8) for the lower body deviation angle sequence Y l Calculate the standard deviation and obtain the standard deviation σ2 of the inclination angle of the lower body.

5. The method according to claim 1, characterized in that: The standard deviation σ3 of the knee bending angle is calculated in (4b3) as follows: (4b3.1) During the pull-up process, the MediaPipe 3D skeleton estimation algorithm is used in real time to extract the waist, knee and ankle key points of each frame of the video. Suppose the 3D coordinates of the waist at time t are The three-dimensional coordinates of the knee are The three-dimensional coordinates of the ankle are (4b3.2) Connect the key points of the waist, knees, and ankles to form a triangle, calculate the three sides of the triangle, and assume that the distance from the waist to the knee at time t is The distance from waist to ankle is The distance from knee to ankle is The calculation formula is as follows: (4b3.3) Let the knee bending angle at time t be According to the side length of the triangle formed by the waist, knee and ankle key points and the cosine theorem of the triangle, the knee bending angle is calculated. The cosine of : (4b3.4) Calculate the knee bending angle using the arccosine function (4b3.5) Calculate the knee bending angle at each moment during the pull-up movement Get the knee bending angle sequence Y k ; (4b3.6) for the knee bending angle sequence Y k Calculate the standard deviation and obtain the standard deviation of the knee bending angle σ3.

6. The method according to claim 1, characterized in that The angle of the elbow joint of the test person at time t during the descent of the pull-up action is calculated in (4c). The implementation is as follows: (4c1) Use the MediaPipe 3D skeleton estimation algorithm to extract the key points of the shoulder, elbow, and wrist. Suppose the 3D coordinates of the shoulder at time t are The three-dimensional coordinates of the elbow are The three-dimensional coordinates of the wrist key points are (4c2) Connect the key points of the shoulder, elbow, and wrist to form a triangle, calculate the three sides of the triangle, and assume that the distance from the shoulder to the elbow at time t is The distance from shoulder to wrist is The distance from elbow to wrist is The calculation formula is as follows: (4c3) Assume that the elbow bending angle is According to the side length of the triangle formed by the shoulder, elbow and wrist key points and the cosine theorem of the triangle, calculate the elbow bending angle The cosine of : (4c4) Calculate the elbow bending angle using the arccosine function

Images