Sequential action segmentation method of athlete skeleton points based on computer vision

Abstract

The invention belongs to the technical field of video processing and action analysis, and discloses a time sequence action segmentation method for athlete skeleton points based on computer vision. An athlete training video is acquired only through a single image acquisition system, the video of an athlete in training is analyzed, key skeleton point coordinate information is extracted by means of a deep learning model, and kinematics characteristics such as angle, angular velocity and angular acceleration are combined, so that action segmentation is completed, training data are generated, and subsequent analysis is performed. The method has the advantages of high efficiency, accuracy and high applicability; and the skilled actions of the athletes are accurately segmented and comprehensively analyzed to generate motion characteristic data of each stage, so that optimized guidance is provided for coaches and the athletes.

Description

Technical Field

[0001] The invention mainly relates to the technical field of video processing and motion analysis, and in particular to a method for segmenting the temporal motion of athlete skeleton points based on computer vision. Background Art

[0002] Periodic motion refers to sports in which movements are repeated continuously. Take rowing as an example. Rowing is a water sport with a long history and is highly dependent on technology. Athletes need to complete the paddling process through coordinated body movements, and their sports technique and posture have a decisive influence on their performance in the competition. Traditional rowing motion analysis usually relies on sensor acquisition, such as wearable devices or motion analysis laboratories. Although these methods have certain advantages in accuracy, their equipment is expensive, the experimental conditions are strict, and they interfere with the natural movement state of athletes.

[0003] With the development of computer vision and deep learning technology, vision-based motion analysis methods have gradually become a research hotspot. These methods do not require complex sensor equipment and can achieve motion trajectory and technical analysis by relying only on video acquisition and key point detection algorithms. However, there are still some technical difficulties in the current vision-based motion segmentation and analysis methods. Taking rowing as an example, there are the following problems: the accurate detection of action switching points depends on high-precision joint point trajectory extraction; the segmentation accuracy is easily affected by noise data and external environments such as lighting and viewing angle; the action stage segmentation is too dependent on single angle information and lacks a calibration mechanism for comprehensive dynamic conditions.

[0004] Therefore, developing a computer vision-based temporal motion segmentation method for athlete skeleton points to achieve accurate and automated motion phase recognition and technical data analysis has important research significance and practical application value. Summary of the invention

[0005] The purpose of the present invention is to provide a method for temporal action segmentation of athlete skeleton points based on computer vision to solve the problems of inaccurate action cutting points, large data noise interference and single segmentation conditions in the prior art.

[0006] To achieve the above object, the present invention provides a method for segmenting the temporal motion of athlete skeleton points based on computer vision, comprising the following steps: Step 1: Data acquisition and input: Use an image acquisition device to collect the athlete's training video and input the video frame containing the skeleton point data; Step 2: Preprocess the video frames containing skeleton point data: Use filtering and denoising algorithms to denoise and smooth the collected original video frames containing skeleton point data to reduce the noise and environmental noise interference of the training video; Step 3: Calculate the timing relationship of the skeleton key point joints: S1: Skeleton key point detection: obtain the human body detection frame in the denoised and smoothed training video, and then extract the coordinates of the 2D and 3D COCO human skeleton key points; S2: Data storage and conversion: extract and store the skeleton key point information in the skeleton key point coordinates;

[0007] Step 4: Calculate the values ​​of the joint relationship of the key points of the skeleton: calculate the angle, angular velocity, and angular acceleration of the key points of the skeleton; Step 5: Action segmentation and cutting point detection: S3: Preliminary detection of action cutting points based on bone angle changes; S4: Introducing angular velocity and angular acceleration, and combining the action stage classification results to further verify and refine the cutting points; S5: Set a time threshold. If the interval between the cutting points is greater than the set time threshold, mark the stage switching point near the minimum and maximum values ​​of the angular velocity, and confirm the switching point through the frame number points where the acceleration changes significantly; S6: If the interval between the cutting points does not exceed the set time threshold, continue to detect the next video frame containing skeleton point data; Step 6: Record the switching points and mark the action phases; Step 7: Dynamically adjust the switching conditions: Combine the angle, angular velocity and angular acceleration, and the overall displacement trend of the skeleton center point to dynamically optimize the cutting point detection conditions; Step 8: Sliding Window Optimization: S7: Introduce a time window smoothing algorithm to perform sliding window filtering on the input video frames containing skeleton point data to reduce erroneous cutting caused by short-term fluctuations; S8: Add time interval constraints to the switching conditions of each action stage to avoid too dense cutting points; Step 9: Data analysis and result output: According to the cutting action phase, calculate the motion indexes in the training process, draw the angle, angular velocity and angular acceleration curves of each phase, and conduct data comparison and optimization analysis on the technical actions of the athletes; If all video frames containing skeleton point data have been processed, the process ends; if not all video frames containing skeleton point data have been processed, the process returns to step 1 and starts again.

[0008] Furthermore, the key skeleton point detection step in step S1 is as follows: the denoised and smoothed training video is input into the RTMDet model to obtain the human body detection frame, and then the two-dimensional and three-dimensional COCO human body key skeleton point coordinates are extracted through the RTMPose model.

[0009] Furthermore, in step S1, the number of key skeleton points of the human body is 17, namely nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0010] Furthermore, in step S2 data storage and conversion, key bone point information in the key bone point coordinates is extracted from the JSON file and stored as a CSV file.

[0011] Furthermore, step 4 calculates the values ​​of the joint relationships of the key skeletal points, and the angular velocity and angular acceleration are calculated by discrete difference formulas. The angle, angular velocity and angular acceleration data are processed by a Gaussian smoothing algorithm to reduce noise interference.

[0012] Furthermore, the angle calculation in step 4 calculates the angle of a specific joint through the set relationship of the bone point coordinates, and the calculation formula for the angular velocity is: , the calculation formula of angular acceleration is ,in is the joint angle, is the angular velocity, is the angular acceleration, is the inter-frame time of the video frames containing the skeleton point data.

[0013] Furthermore, the knee joint angle is calculated from the three-point coordinates of the hip joint, knee joint and ankle, and the hip joint angle is calculated from the three-point coordinates of the shoulder joint, hip joint and knee joint.

[0014] Furthermore, the sports indicators in the training process in step 9 include the stroke frequency, the number of strokes, the push-pull ratio in the action segmentation stage, the angle, angular velocity, and angular velocity acceleration curve at any time in the action stage, and the comparative analysis results of the athletes' technical movements.

[0015] Furthermore, the method also includes step 10, which is to optimize the athletes' technical movements and develop personalized training plans for the athletes based on data analysis and result output.

[0016] Beneficial effects: The present invention provides a method for temporal motion segmentation of athlete skeleton points based on computer vision to solve the problems of inaccurate action cutting points, large data noise interference, and single segmentation conditions in the prior art. The method of the present invention does not require the use of additional sensors. It only collects athlete training videos through a single image acquisition system. By analyzing the videos of athletes in training, the coordinate information of key skeleton points is extracted with the help of a deep learning model, and kinematic features such as angles, angular velocities, and angular accelerations are combined to complete motion segmentation, generate training data, and conduct subsequent analysis. This method has the advantages of high efficiency, accuracy, and strong applicability. It can be applied to periodic sports, including but not limited to rowing, running, cycling, etc. It can accurately segment and comprehensively analyze the technical movements of athletes, generate motion feature data for each stage, and provide optimized guidance for coaches and athletes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is an overall flow chart of a method for segmenting athlete skeleton points based on computer vision according to an embodiment of the present invention; Figure 2 1 is a schematic diagram of key point detection of human skeleton in a method for segmenting athlete skeleton points based on computer vision according to an embodiment of the present invention; Figure 3 1 is a segmentation schematic diagram of a method for segmenting a time sequence action of a player's skeleton points based on computer vision according to an embodiment of the present invention; Figure 4 It is a normalized one-paddle interval comparison diagram of the full-course training knee joint angle curve of the time-series action segmentation method of the athlete's skeleton points based on computer vision involved in an embodiment of the present invention; Figure 5 It is a normalized one-paddle interval comparison diagram of the full-course training knee joint angular velocity curve of the time-series motion segmentation method of the athlete's skeleton points based on computer vision involved in an embodiment of the present invention; Figure 6 It is an example diagram of time series curves of knee joint angle, angular velocity and angular acceleration of the method for time series motion segmentation of athlete skeleton points based on computer vision according to an embodiment of the present invention; Figure 7 It is a comparative bar chart of different athlete training indicators of the method for temporal action segmentation of athlete skeleton points based on computer vision involved in an embodiment of the present invention.

[0018] Description of reference numerals: 0 is the nose; 1 is the left eye; 2 is the right eye; 3 is the left ear; 4 is the right ear; 5 is the left shoulder; 6 is the right shoulder; 7 is the left elbow; 8 is the right elbow; 9 is the left wrist; 10 is the right wrist; 11 is the left hip; 12 is the right hip; 13 is the left knee; 14 is the right knee; 15 is the left ankle; 16 is the right ankle. DETAILED DESCRIPTION

[0019] The preferred mechanism and implementation method of the present invention are further described below in conjunction with the accompanying drawings and specific implementation methods.

[0020] Figures 1 to 7 As shown, the embodiment of the present invention discloses a technical solution for a method for segmenting the temporal actions of athlete skeleton points based on computer vision. Figure 1 is an overall flow chart of a method for segmenting athlete skeleton points based on computer vision in a temporal sequence of motions according to an embodiment of the present invention. Figure 2 Schematic diagram of key point detection of human skeleton in a method for segmenting athlete skeleton points based on computer vision according to an embodiment of the present invention Example

[0021] The athletes involved in the embodiments of the present invention are rowing athletes.

[0022] A method for segmenting the temporal motion of athlete skeleton points based on computer vision, characterized in that it comprises the following steps: Step 1: (1) Data acquisition: Use a single-camera device with a high frame rate (e.g., 30 fps) to collect land or water training videos of rowers. The camera is set in a fixed position to ensure that the athlete's entire body is captured and the complete movement trajectory is covered. During video acquisition, background interference, such as other people or equipment entering the shooting screen, is minimized.

[0023] (2) Decompose the training video into image sequences at a fixed frame rate to ensure that each frame can continuously capture the athlete's movements and input video frames containing skeleton point data.

[0024] Step 2: Preprocess the video frames containing skeleton point data: (3) Use Gaussian filtering or median filtering to reduce the noise of the training video; (4) Use image enhancement technology to optimize the brightness and contrast of training videos and improve visual effects; Step 3: Key point detection: Convert the training video to a standard format that is compatible with the RTMDet and RTMPose models, such as MP4 or JPEG. Use the trained RTMDet model to identify the overall bounding box of the athlete in the video, crop the detected bounding box and input it into the RTMPose model to detect the key bone points of the human body. The RTMPose model outputs the two-dimensional (2D) and three-dimensional (3D) coordinates of 17 key bone points, including: Head: nose, left eye, right eye, left ear, right ear.

[0025] Upper limbs: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist.

[0026] Trunk and lower extremities: left hip, right hip, left knee, right knee, left ankle, right ankle.

[0027] Step 4: Data storage and conversion: Save the coordinates of the bone points output by the model in JSON format, extract the coordinate data of key points related to motion segmentation (such as knee joints, hip joints, and shoulder joints), organize the extracted data and store it as a CSV file in the following format: Each row represents the skeleton point information of one frame.

[0028] Each column represents the X, Y, and Z coordinates of a bone point.

[0029] Step 5: Angle, angular velocity, angular acceleration and smoothing: Calculate the angle, angular velocity and angular acceleration of some key bone points (such as knee joint, hip joint, shoulder joint, etc.). The angular velocity and angular acceleration are calculated using discrete difference formulas, and the angle, angular velocity and angular acceleration data are processed using Gaussian smoothing algorithm to reduce noise interference.

[0030] Target joint selection: The knee, hip and shoulder joints were selected as the main research objects; (5) Angle formula: Calculate the joint angle based on the coordinates of adjacent bone points, for example: Knee joint angle: Based on the coordinates of the hip joint, knee joint, and ankle joint, the vector angle formula is used to calculate: ,in, and are the vectors from the hip joint to the knee joint and from the knee joint to the ankle joint; (6) The angular velocity is calculated using the discrete difference formula: ,in, and They are the angle values ​​of the tth frame and the t+1th frame respectively.

[0031] (7) The angular acceleration is calculated using the second-order difference formula: .

[0032] (8) Smoothing: Gaussian filtering is used to smooth the angle, angular velocity, and angular acceleration data to reduce noise interference. The smoothing formula is: ; Sliding window method: Set a fixed window (such as 5 frames) to perform mean smoothing on the data.

[0033] Step 6: Preliminary segmentation based on angle changes: (9) Set the preliminary cutting point detection rules: When the knee joint angle drops rapidly from the maximum to the minimum, it is marked as the end point of "rowing"; When the knee angle rises rapidly from the minimum to the maximum, it is marked as the starting point of "rowing".

[0034] (10) Record the frame number corresponding to each cutting point; (11) Store the preliminary segmentation results as a segmentation point list for subsequent optimization.

[0035] Step 7: Cut point verification and refinement: (12) Introduce angular velocity and angular acceleration to verify the cutting point; (13) Detecting the cutting point near the extreme point (maximum or minimum) of the angular velocity; (14) Use significant changes in angular acceleration (such as sign reversal) to confirm the cutting point; (15) Skeleton center point displacement detection; (16) Calculate the overall displacement trend of the athlete's skeleton (such as forward or backward during paddling); (17) Use displacement trends to further verify or refine the cutting points; (18) Dynamic adjustment of cutting conditions: (19) Combine angle, angular velocity, and angular acceleration to define more dynamic action switching conditions. For example, when the angular velocity and angular acceleration change significantly at the same time, the cutting point is marked first to avoid misjudgment caused by short-term fluctuations.

[0036] Step 8: Time constraints and deduplication processing: (20) Introducing time window smoothing mechanism: Set a fixed time window (such as 0.5 seconds) to smooth the preliminary cutting point list; add time interval restrictions to avoid too dense cutting points.

[0037] (21) Deduplication processing: merge cutting points with too short intervals into a single cutting point to generate the final cutting point list.

[0038] Step 9: Training metric calculation and visualization The following metrics are calculated: Stroke rate: The number of strokes per minute is calculated based on the time interval between cut points.

[0039] Number of paddles: The number of planned paddle cycles.

[0040] Push-pull ratio: the ratio of time spent pushing the paddle to that spent pulling the paddle.

[0041] Data visualization: draw time series graphs of knee joint angle, angular velocity, and angular acceleration; generate visualization reports of athletes’ training status.

[0042] Figure 3In the figure, the horizontal axis is time and the vertical axis is angle, showing the segmentation diagram of each paddle. The black dashed line interval is the knee joint angle curve of one paddle, and the red dashed line indicates the switching point between pulling and returning the paddle. The front section is pulling and the rear section is returning the paddle. At the beginning of one paddle, the pulling angle increases from the minimum 35 degrees to the peak of 165 degrees, reaching the action switching point. After that, the action is converted to returning the paddle and slowly decreases to the lowest point of 35 degrees, and the one paddle movement cycle ends.

[0043] Figure 4 In the figure, the horizontal axis shows time and the vertical axis shows angle. The knee joint angle curve of each stroke in the whole exercise process is normalized to the same time dimension from 0 seconds to the end of the rowing cycle. It can be seen that the duration of each stroke is about 2.2 seconds and the angle peak is about 165 degrees, which means that the angle curve of each rowing cycle in the whole 2km exercise process is normalized. The angle curves of all rowing cycles are moved to the same time dimension, showing the accuracy of the segmentation results and the stability of the movement.

[0044] Figure 5 In the figure, the horizontal axis shows time and the vertical axis shows angular velocity. The knee joint angular velocity curve of each stroke in the whole exercise process is normalized to the same time dimension from 0 seconds to the end of the rowing cycle. It can be seen that the duration of each stroke is about 2.2 seconds, the peak angular velocity of the pull-stroke is about 300 (degrees / s), and the peak angular velocity of the return stroke is about 220 (degrees / s), which means that the angular velocity curve of each rowing cycle in the whole 2km exercise process is normalized, and the angular velocity curves of all rowing cycles are moved to the same time dimension.

[0045] Figure 6 In the figure, a time series curve example of the knee joint angle, angular velocity and angular acceleration is drawn. The horizontal axis is time, and the vertical axis is angle, angular velocity, angular acceleration, etc., which shows the transformation of the knee joint angle, angular velocity and angular acceleration in the time series, and shows the changes of the athlete's knee joint angle, angular velocity and angular acceleration in the interval of 50 to 60 seconds of exercise.

[0046] Figure 7 , represents the comparison results of various detailed data of 2 km obtained by two different athletes after the segmentation algorithm, mainly the comparison results of the number of oars, average rowing time, average rowing frequency, and push-pull ratio. It represents the comparison results of various detailed data obtained by athlete A and athlete B after the whole exercise by segmentation algorithm. Among them, the comparison of the number of oars is 208 (times): 190 (times), the comparison of the average rowing time is 1.64 (s): 2.04 (s), the comparison of the average rowing frequency is 36 (oars / minute): 30 (oars / minute), and the push-pull ratios are 49.878%: 50.122%, 49.870%: 50.130%.

[0047] The present invention provides a method for segmenting the temporal motion of athlete skeleton points based on computer vision to solve the problems of inaccurate motion cutting points, large data noise interference, and single segmentation conditions in the prior art. The method of the present invention does not need to use additional sensors, and only collects athlete training videos through a single image acquisition system. By analyzing the video of rowing athletes in training, the coordinate information of key skeleton points is extracted with the help of a deep learning model, and kinematic features such as angles, angular velocities, and angular accelerations are combined to complete motion segmentation, generate training data, and perform subsequent analysis. This method has the advantages of high efficiency, accuracy, and strong applicability. It is suitable for periodic sports events, including but not limited to rowing, running, cycling, etc., and accurately segments and comprehensively analyzes the technical movements of athletes, generates motion feature data at each stage, and provides optimized guidance for coaches and athletes.

[0048] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. However, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for segmenting athlete skeleton points in time sequence based on computer vision, characterized in that: The following steps are involved: Step 1: Data acquisition and input: Use an image acquisition device to collect the athlete's training video and input the video frame containing the skeleton point data; Step 2: Preprocess the video frames containing skeleton point data: Use filtering and denoising algorithms to denoise and smooth the collected original video frames containing skeleton point data to reduce the noise and environmental noise interference of the training video; Step 3: Calculate the timing relationship of the skeleton key point joints: S1: Skeleton key point detection: obtain the human body detection frame in the denoised and smoothed training video, and then extract the coordinates of the 2D and 3D COCO human skeleton key points; S2: Data storage and conversion: extract and store the skeleton key point information in the skeleton key point coordinates; Step 4: Calculate the values ​​of the joint relationship of the key points of the skeleton: calculate the angle, angular velocity, and angular acceleration of the key points of the skeleton; Step 5: Action segmentation and cutting point detection: S3: Preliminary detection of action cutting points based on bone angle changes; S4: Introducing angular velocity and angular acceleration, and combining the action stage classification results to further verify and refine the cutting points; S5: Set a time threshold. If the interval between the cutting points is greater than the set time threshold, mark the stage switching point near the minimum and maximum values ​​of the angular velocity, and confirm the switching point through the frame number points where the acceleration changes significantly; S6: If the interval between the cutting points does not exceed the set time threshold, continue to detect the next video frame containing skeleton point data; Step 6: Record the switching points and mark the action phases; Step 7: Dynamically adjust the switching conditions: Combine the angle, angular velocity and angular acceleration, and the overall displacement trend of the skeleton center point to dynamically optimize the cutting point detection conditions; Step 8: Sliding Window Optimization: S7: Introduce a time window smoothing algorithm to perform sliding window filtering on the input video frames containing skeleton point data to reduce erroneous cutting caused by short-term fluctuations; S8: Add time interval constraints to the switching conditions of each action stage to avoid too dense cutting points; Step 9: Data analysis and result output: According to the cutting action phase, calculate the motion indexes in the training process, draw the angle, angular velocity and angular acceleration curves of each phase, and conduct data comparison and optimization analysis on the technical actions of the athletes; If all video frames containing skeleton point data have been processed, the process ends; if not all video frames containing skeleton point data have been processed, the process returns to step 1 and starts again.

2. The method for segmenting athlete skeleton points based on computer vision according to claim 1, characterized in that: The steps of key bone point detection in step S1 are: input the denoised and smoothed training video into the RTMDet model to obtain the human body detection frame, and then extract the two-dimensional and three-dimensional COCO human body key bone point coordinates through the RTMPose model.

3. The method for segmenting the temporal motion of athlete skeleton points based on computer vision according to claim 2, characterized in that: In step S1, there are 17 key bone points of the human body, namely nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

4. The method for segmenting athlete skeleton points based on computer vision according to claim 1, characterized in that: In step S2 data storage and conversion, the key bone point information in the key bone point coordinates is extracted from the JSON file and stored as a CSV file.

5. The method for segmenting athlete skeleton points based on computer vision according to claim 1, characterized in that: Step 4 calculates the joint relationship values ​​of key skeletal points. The angular velocity and angular acceleration are calculated using discrete difference formulas. The angle, angular velocity, and angular acceleration data are processed using a Gaussian smoothing algorithm to reduce noise interference.

6. The method for segmenting the temporal motion of athlete skeleton points based on computer vision according to claim 1, characterized in that: The angle calculation in step 4 calculates the angle of a specific joint through the set relationship of the bone point coordinates. The calculation formula for the angular velocity is: , the calculation formula of angular acceleration is ,in is the joint angle, is the angular velocity, is the angular acceleration, is the inter-frame time of the video frames containing the skeleton point data.

7. The method for segmenting athlete skeleton points based on computer vision according to claim 6, characterized in that: The knee joint angle is calculated from the three-point coordinates of the hip joint, knee joint, and ankle, and the hip joint angle is calculated from the three-point coordinates of the shoulder joint, hip joint, and knee joint.

8. The method for segmenting athlete skeleton points based on computer vision according to claim 1, characterized in that: The sports indicators in the training process in step 9 include the stroke frequency, the number of strokes, the push-pull ratio in the action segmentation phase, the angle, angular velocity, the angular velocity acceleration curve at any time in the action phase, and the comparative analysis results of the athletes' technical movements.

9. The method for segmenting athlete skeleton points based on computer vision according to claim 1, characterized in that: The method also includes step 10, which is to optimize the athletes' technical movements and develop personalized training plans for the athletes based on data analysis and result output.

Images