Method for recognizing pattern skipping rope action and intelligent judging based on visual space-time feature matching

Abstract

The application provides a pattern skipping rope action recognition and intelligent judging method based on visual space-time feature matching, and belongs to the technical field of visual information processing. The first step is multi-view video data acquisition and synchronization; the second step is video preprocessing and frame extraction; the third step is skeleton key point extraction based on improved YOLOv8-pose; the fourth step is skeleton key point sequence post-processing and kinematic feature extraction; the fifth step is rope body motion trajectory extraction; the sixth step is space-time feature modeling and action sequence recognition; the seventh step is action quality evaluation and intelligent judging score; the eighth step is system deployment and real-time inference. The application effectively enhances the recognition ability of the complex action of the pattern skipping rope by fusing YOLOv8-pose and CA attention mechanism and introducing a space-time feature matching module, and has higher precision in distinguishing similar actions such as cross jumping and side swing crossing.

Description

Technical Field

[0001] This invention provides a method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching, belonging to the field of visual information processing technology. Background Technology

[0002] In recent years, vision-based human pose estimation and action recognition technologies have made significant progress, and their applications have gradually expanded into the sports field.

[0003] Due to its high level of skill, artistry, and competitiveness, synchronized rope skipping has developed rapidly in international and domestic competitions in recent years, posing unprecedented challenges to referees. Traditional manual scoring methods rely heavily on the experience and instantaneous judgment of referees. In identifying high-speed, complex, and varied technical movements, these methods inevitably suffer from strong subjectivity, difficulty in ensuring consistency, high labor costs, and fatigue. Especially in large-scale competitions, the accurate classification and evaluation of various rotations, crosses, somersaults, and other synchronized movements performed by athletes requires more objective and efficient technical support. Currently, AI-based referee-assisted technology has been explored in sports such as gymnastics and diving, but in the unique field of synchronized rope skipping, which combines multiple information such as rhythm, spatial trajectory, and body posture, mature and reliable automated solutions are still lacking.

[0004] Patent document CN202510951955 discloses a system and method for recognizing movements in freestyle rope skipping. It identifies movements by cross-mapping the trend of joint angle changes with the switching of rope direction, outputting basic state labels such as "take-off," "leap," and "landing," as well as distinctions between major movement categories such as "single-leg jump," "double-leg jump," "cross-step jump," and "double unders." However, for movement variations that require precise differentiation in freestyle rope skipping competitions (such as transitional combinations between cross jumps and side swings, and between single and double unders), its recognition granularity is insufficient to support the demand for fine-grained movement classification, making it difficult to meet the standards for detailed movement type subdivision in competition-level movement judging.

[0005] Vision-based intelligent sports judging systems possess practical feasibility and widespread application value. However, existing systems primarily focus on counting and simple violation identification, lacking the capability for precise recognition and professional scoring of complex combinations of movements in freestyle rope skipping. Therefore, developing an AI system capable of intelligently sensing, understanding, and judging freestyle rope skipping movements has significant application value and broad prospects. Summary of the Invention

[0006] The objective of this invention is to provide a method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching, specifically covering the following aspects:

[0007] It can accurately identify common fancy moves such as single jump, double jump, triple jump, cross jump, scramble, and side swing cross, and precisely locate the start and end time boundaries of the moves.

[0008] Based on the analysis of single-frame pose information, we further capture the evolution of action sequences in the time dimension and extract discriminative spatiotemporal feature representations.

[0009] Based on the action recognition results, the system automatically completes action counting, difficulty level mapping, quality assessment, and comprehensive scoring, achieving end-to-end automated processing from video input to score output.

[0010] The proposed solution should be able to run in real time under normal competition conditions, have reasonable hardware dependencies, and be suitable for practical deployment.

[0011] The specific technical solution of this invention:

[0012] A method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching includes the following steps:

[0013] Step 1: Multi-view video data acquisition and synchronization;

[0014] Simultaneous video capture of synchronized rope skipping athletes is conducted using a camera system with at least two viewing angles, with two video streams respectively. and The two cameras are synchronized via hardware triggering or software synchronization based on audio feature alignment. The video capture frame rate is set to... FPS. The captured motion types cover single jump, double jump, triple jump, cross jump, choreographed jump, side swing cross, and combination moves.

[0015] Step 2: Video preprocessing and frame extraction;

[0016] The acquired video stream and Image frames are extracted at fixed frame intervals to form an image sequence. The following preprocessing operations are performed sequentially, among which Indicates the viewpoint number. Indicates the frame number corresponding to the timestamp:

[0017] Scale the image to a uniform width and height. The number of pixels is determined to meet the input requirements of the subsequent YOLOv8-pose model. Gaussian filtering or median filtering is used to remove random noise from the image. The preprocessed image sequence is denoted as... .

[0018] Step 3: Extraction of skeletal key points based on the improved YOLOv8-pose;

[0019] For each frame of the preprocessed image An improved YOLOv8-pose model was used for human detection and skeletal key point extraction.

[0020] YOLOv8-pose extends the pose estimation branch on the basis of the YOLOv8 object detection framework, and outputs the human bounding box and the coordinates of 17 skeletal key points simultaneously in a single forward propagation.

[0021] A CA attention mechanism is embedded in the C2f module of the YOLOv8 backbone network. Channel attention is decomposed into two one-dimensional feature encoding processes, which aggregate features along the horizontal and vertical directions respectively to capture precise location information.

[0022] Step 4: Post-processing of skeletal keypoint sequences and extraction of kinematic features;

[0023] The fused skeletal keypoint sequence Post-processing and kinematic feature extraction are performed. ;

[0024] Specifically, it includes:

[0025] (1) Temporal smoothing filter

[0026] A Kalman filter is used to smooth the two-dimensional trajectory of each keypoint. Through prediction-update iteration, the smoothed keypoint coordinate sequence is obtained. ;

[0027] (2) Interpolation of missing key points

[0028] For keypoints marked as missing, cubic spline interpolation is used to complete them; let the coordinates of the known keypoints before and after the missing segment be... and Construct a piecewise cubic polynomial that satisfies the boundary second-order continuity for interpolation;

[0029] (3) Calculation of kinematic characteristics

[0030] Based on the smoothed keypoint sequence, kinematic features are calculated for subsequent action recognition and quality assessment.

[0031] Step 5: Extracting the trajectory of the rope's motion;

[0032] (1) Rope body region segmentation

[0033] Preprocessed image frames In this study, the rope region is extracted using color features or a lightweight semantic segmentation network. The HSV color space thresholding method is employed. Let the HSV value of each image pixel be... Rope mask Determined by the following formula:

[0034] ,

[0035] in The threshold parameters are preset based on the rope color.

[0036] (2) Positioning of the end point of the rope

[0037] Morphological refinement of the rope mask is performed to extract the rope's skeleton lines. The rope's end points are determined based on their distance from the wrist key point: the end closer to the wrist key point is the handle end, and the end farther away is the free end, i.e., the rope's end.

[0038] Let the set of skeleton line endpoints be The coordinates of the key points on the wrist are and , the end point of the rope body Selected as:

[0039] ,

[0040] That is, select the skeletal end closest to any wrist as the end of the rope.

[0041] (3) Extraction of rope motion features

[0042] Coordinates of the end of the rope in consecutive frames Extract the following features:

[0043] Rope swing direction: Defined as the end of the rope in the horizontal plane ( The direction angle of motion (in a plane):

[0044] ,

[0045] Direction switching event: When the change in direction angle between two adjacent frames exceeds a preset threshold. When the angle is 90°, mark the current frame as the direction switching frame:

[0046] ,

[0047] Rope trajectory curvature: The local curvature of the rope's end trajectory fitted within a sliding window, reflecting the smoothness of the rope's motion. For three points within the window... The curvature is approximately:

[0048] ;

[0049] Step 6: Spatiotemporal feature modeling and action sequence recognition;

[0050] After acquiring the skeletal keypoint sequence, kinematic features, and rope motion features, a spatiotemporal feature representation is constructed and action recognition is performed.

[0051] (1) Construction of skeletal spatiotemporal map

[0052] Modeling the human skeletal sequence as a spatiotemporal graph The set of nodes Indicates the first The first frame A joint. Edge set. It consists of two parts:

[0053] Space edge Based on the natural connection relationships of the human skeleton, connect adjacent joints within the same frame;

[0054] Time edge : Connecting corresponding nodes of the same joint in adjacent frames, i.e. and The edges between them.

[0055] Each node The feature vector is composed of the joint's coordinates, velocity, acceleration, and the corresponding rope state.

[0056] (2) Spatiotemporal Graph Convolutional Network

[0057] An improved spatiotemporal graph convolutional network is used to extract features from the skeletal spatiotemporal graph. For nodes... Its neighborhood includes the spatial neighborhood. and time neighborhood The graph convolution operation is defined as follows:

[0058] ,

[0059] in and These are the input and output features, respectively. and The learnable spatial and temporal convolution weight matrix. and This is the normalization factor.

[0060] A multi-scale temporal convolution module is introduced into the network, employing dilated convolutions with different dilation rates in parallel processing to capture action patterns across different time spans:

[0061] ,

[0062] in For the set of expansion rates, For the first The expansion step size of each branch, This represents the number of branches.

[0063] (3) Integration of rope features

[0064] Rope motion features are fused with temporal features as an auxiliary modality. A cross-modal attention mechanism is employed, using skeletal features as the query and rope features as the key and value, to calculate the enhancement of the skeletal representation by rope information.

[0065] ,

[0066] in , , , and These are sequences of skeletal features and rope features, respectively.

[0067] The fused features are mapped to an action category probability distribution through a fully connected layer.

[0068] (4) Temporal action segmentation and recognition

[0069] Sliding window processing is applied to the video stream, with a window length of [missing information]. The duration is set according to the expected action duration. Within each window, the model outputs an action class probability vector. ,in This represents the number of action categories.

[0070] To accurately locate the start and end boundaries of the action, a dynamic time warping algorithm is used to align the predicted action sequence with a standard action template. Let the predicted sequence be... The template sequence is Both are sequences of skeletal keypoints. A distance matrix is ​​defined. ,in DTW solves for the optimal alignment path using dynamic programming, and the cumulative distance matrix is ​​calculated as follows:

[0071] ,

[0072] The alignment point corresponding to the optimal path is the start and end boundary of the action.

[0073] Step 7: Action quality assessment and intelligent referee scoring;

[0074] After motion recognition is completed, the process moves to the intelligent referee scoring stage. The scoring system covers difficulty assessment, completion quality evaluation, violation detection, and comprehensive scoring.

[0075] (1) A database of standard movements for fancy rope skipping was established.

[0076] (2) Action difficulty level mapping

[0077] Establish a mapping table between action labels and difficulty levels. The difficulty level of a basic action is denoted as... Other moves are assigned a relative difficulty level based on competition rules and expert experience. :

[0078] Total difficulty score of a performance The sum of the difficulty coefficients of all identified valid actions:

[0079] , in For the first The number of times a type of action is effectively completed.

[0080] (2) Complete the quality assessment

[0081] For each identified action, the quality is quantified by comparing the deviation between the actual skeletal trajectory and the standard action template.

[0082] Set action The standard template is a skeletal keypoint sequence. The actual detected sequence is (The lengths are consistent after DTW alignment). Define the joint. At any moment Deviation:

[0083] ,

[0084] Considering the varying importance of different joints during movement, joint weights are introduced. The normalized overall bias of the action is:

[0085] ,

[0086] The deviation is converted into a quality score using a preset mapping function. :

[0087] ,

[0088] in The sensitivity coefficient controls the steepness of the score's response to bias.

[0089] Overall performance quality score The weighted average of the quality scores for all movements:

[0090] ,

[0091] in The difficulty coefficient of the move is used to give greater weight to the quality of high-difficulty moves in the total score.

[0092] (3) Detection of violations

[0093] Establish a violation rule database to automatically identify and deduct points for violations. This is the sum of the points deducted for each violation.

[0094] (4) Calculation of comprehensive score

[0095] The final score is calculated by combining the difficulty score, the quality score, and the penalty points for violations:

[0096] ,

[0097] in This is the total difficulty score, the raw, unscaled value. The normalized quality score; Points will be deducted for violations.

[0098] Step 8: System Deployment and Real-time Inference;

[0099] Deploy the trained model to a real-world application environment. Perform the following optimizations:

[0100] (1) Model quantization and acceleration

[0101] TensorRT or ONNX Runtime is used to accelerate model inference. INT8 quantization is performed on the YOLOv8-pose part, significantly improving inference speed while maintaining controllable accuracy loss. The quantization process converts floating-point weights... Mapped to integer :

[0102] ,

[0103] in To quantize the step size, This is the zero-point offset. The inverse quantization formula is: .

[0104] (2) Multithreaded pipeline design

[0105] Video acquisition, preprocessing, model inference, postprocessing, and scoring calculation are all placed in independent threads, and data is transferred through a shared memory queue, making full use of multi-core CPU / GPU resources and reducing end-to-end latency.

[0106] (3) Visual output

[0107] It displays real-time skeletal point connections, current action label, score, remaining time, and violation alerts. The visualization interface is built based on the OpenCV or Qt framework and overlays original video frames.

[0108] The beneficial effects of the technical solution of this invention are as follows:

[0109] By integrating YOLOv8-pose with the CA attention mechanism and introducing a spatiotemporal feature matching module, the ability to recognize complex movements in fancy rope skipping is effectively enhanced, especially in distinguishing similar movements such as cross jumps and side swings.

[0110] It provides a complete processing chain from video acquisition, pose estimation, action recognition to difficulty determination, quality assessment, and comprehensive scoring, filling the gap in professional scoring functions in existing systems.

[0111] It not only utilizes single-frame pose information, but also captures the dynamic evolution of action sequences through temporal modeling, significantly improving the robustness of continuous high-frequency action recognition and the accuracy of temporal segmentation.

[0112] The solution is built on a mature open-source framework and a general-purpose hardware platform, and has good real-time processing capabilities and deployment feasibility in both regular competitions and teaching and training scenarios. Attached Figure Description

[0113] Figure 1 This is a flowchart of the present invention;

[0114] Figure 2 The layout of the multi-view camera device of the present invention;

[0115] Figure 3 This is a schematic diagram of the human body boundary frame and 17 key points of the present invention. Detailed Implementation

[0116] The overall technical process of the fancy rope skipping motion recognition and intelligent refereeing method based on visual spatiotemporal feature matching provided by this invention is as follows: Figure 1 As shown below, the steps will be explained in detail.

[0117] Step 1: Multi-view video data acquisition and synchronization;

[0118] like Figure 2 The system employs cameras with at least two viewing angles to simultaneously capture video of rhythmic rope skipping athletes. A typical configuration includes a frontal main view and a side-front auxiliary view. The purpose of this multi-view setup is to mitigate the failure of key point detection caused by rope or limb obstruction in single-view scenarios, and to improve the continuity and completeness of skeletal key point extraction.

[0119] Let the two video streams be respectively and The two camera systems employ either hardware synchronization triggering or software synchronization methods based on audio feature alignment to ensure that each captured frame has a consistent timestamp reference. The video capture frame rate is set to... The frame rate (fps) is set to ensure that the temporal sampling density for high-frequency movements such as multi-swings meets the analysis requirements. The types of movements collected cover common fancy movements such as single swings, double swings, triple swings, cross jumps, choreographed moves, side swings, and combinations.

[0120] During the data acquisition process, the camera setup must meet the following conditions: the main-view camera should be positioned 3 to 5 meters directly in front of the athlete, with a lens height of approximately 1.2 to 1.5 meters, ensuring the athlete's entire body is within the frame; the auxiliary-view camera should be positioned approximately 45 degrees to the side and front, complementing the main-view camera. A solid-color backdrop or a relatively uniform environment is recommended to reduce the difficulty of subsequent image segmentation and keypoint detection.

[0121] Step 2: Video preprocessing and frame extraction;

[0122] The acquired video stream and Image sequences are extracted at fixed frame intervals. Given sufficient computational resources, a frame-by-frame extraction method is used to preserve complete temporal information.

[0123] Extracted image frames (in Indicates the viewpoint number. The following preprocessing operations are performed sequentially (indicating the frame number corresponding to the timestamp):

[0124] Scale the image to a uniform width and height. The image size is set to pixels to meet the input requirements of the subsequent YOLOv8-pose model. Scaling employs bilinear interpolation to maintain image detail while ensuring computational efficiency. The original image dimensions are set to width and height. Then the scaled pixel coordinates Compared with the original coordinates The relationship is:

[0125] ,

[0126] Gaussian filtering or median filtering is used to remove random noise from the image. The Gaussian filter kernel function is:

[0127] ,

[0128] in The standard deviation is used to control the smoothness.

[0129] The preprocessed image sequence is denoted as .

[0130] Step 3: Extraction of skeletal key points based on the improved YOLOv8-pose;

[0131] For each frame of the preprocessed image An improved YOLOv8-pose model was used for human detection and skeletal keypoint extraction. YOLOv8-pose extends the pose estimation branch on the basis of the YOLOv8 object detection framework, and can simultaneously output the human bounding box and the coordinates of 17 skeletal keypoints in a single forward propagation.

[0132] (1) Network structure improvement - embedding Coordinate Attention mechanism

[0133] To enhance the model's ability to perceive the spatial relationships of joints in fancy rope skipping movements, this invention embeds the Coordinate Attention (CA) mechanism into the C2f module of the YOLOv8 backbone network.

[0134] The core idea of ​​the CA mechanism is to decompose channel attention into two one-dimensional feature encoding processes, aggregating features along the horizontal and vertical directions respectively to capture precise location information. The specific calculation process is as follows:

[0135] Let the input feature map be First along the horizontal direction (width) ) and vertical direction (height) Global average pooling is performed separately to generate feature maps for two directions of perception:

[0136] , ,

[0137] in Encoding features in the height direction, Encoding features in the width direction.

[0138] Will and After being pieced together, it is connected through a shared... Convolution Transform Function Dimensionality reduction and nonlinear activation:

[0139] ,

[0140] in This indicates a splicing operation along a spatial dimension. It is a nonlinear activation function (such as ReLU or SiLU). As an intermediate feature, This is for the dimensionality reduction ratio.

[0141] Then Split along spatial dimensions and Through two convolution and Restore the number of channels and generate attention weights using the Sigmoid function:

[0142] ,

[0143] in For the Sigmoid function, , .

[0144] Finally, the output of the CA module is:

[0145] ,

[0146] This mechanism, by explicitly encoding positional information, enables the model to more accurately capture the relative positional relationships between joints (such as the orientation of the wrist relative to the shoulder, changes in the angle of the knee joint, etc.), and has a significant gain in distinguishing similar movements in fancy rope skipping.

[0147] (2) Key point detection output

[0148] The YOLOv8-pose model outputs the following information for each frame:

[0149] Human body bounding box: ,in Normalized coordinates of the center point The bounding box width and height, , where is the confidence level.

[0150] Skeletal key point coordinates: ,in For the first Normalized coordinates of key points This represents the confidence level at that point.

[0151] The 17 key points include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0152] For multi-view videos, detection is performed for each view to obtain a keypoint sequence. and .like Figure 3 As shown.

[0153] (3) Multi-perspective key point fusion

[0154] The key point detection results from two perspectives at the same time are fused to improve the accuracy and robustness of key point localization.

[0155] First, for both perspectives, the confidence level is greater than the threshold. For key points (such as 0.5), a weighted average method is used for fusion:

[0156] ,

[0157] ,

[0158] If a keypoint meets the confidence standard only in one viewpoint, the detection result of that viewpoint is directly adopted; if it does not meet the standard in both views, the keypoint is marked as missing and subsequently filled in using interpolation methods.

[0159] The principles involved:

[0160] YOLOv8-pose is based on the Anchor-Free detection paradigm. It predicts bounding boxes and keypoint heatmaps separately through decoupled detection heads. Its loss function includes bounding box regression loss (CIoU Loss), keypoint regression loss (OKS Loss), and classification loss.

[0161] Inspired by the spatial attention mechanism in the human visual system, Coordinate Attention improves the accuracy of position-sensitive tasks by decomposing spatial attention into two one-dimensional encoding processes, with almost no increase in computational overhead.

[0162] Step 4: Post-processing of skeletal keypoint sequences and extraction of kinematic features;

[0163] The fused skeletal keypoint sequence ( Post-processing and kinematic feature extraction are then performed.

[0164] (1) Temporal smoothing filter

[0165] The original keypoint coordinates are jittery due to the detection noise. A Kalman filter is used to smooth the two-dimensional trajectory of each keypoint.

[0166] The state equation and observation equation for the Kalman filter are as follows:

[0167] ,

[0168] ,

[0169] Wherein, the state vector Includes position and velocity; state transition matrix Uniform motion model adopted; observation matrix Extract only the location component; The process noise covariance matrix is... To observe the noise covariance matrix.

[0170] The smoothed keypoint coordinate sequence is obtained through prediction-update iteration. .

[0171] (2) Interpolation of missing key points

[0172] For keypoints marked as missing, cubic spline interpolation is used to complete them. Let the coordinates of the known keypoints before and after the missing segment be... and We construct a piecewise cubic polynomial that satisfies the boundary condition of second-order continuity for interpolation.

[0173] (3) Calculation of kinematic characteristics

[0174] Based on the smoothed keypoint sequence, the following kinematic features are calculated for subsequent action recognition and quality assessment:

[0175] Joint angle: for three adjacent key points (e.g., shoulder-elbow-wrist), joint angles Defined as a vector and The included angle:

[0176] ,

[0177] Key joint angles include: left and right shoulder-elbow angles, left and right elbow-wrist angles, left and right hip-knee angles, and left and right knee-ankle angles.

[0178] Joint angular velocity: approximated by the angular difference values ​​between adjacent frames.

[0179] ,

[0180] in This is the frame interval time.

[0181] Key point movement speed: 1st A key point at a moment velocity vector:

[0182] ,

[0183] Overall exercise intensity: The weighted norm of the movement speed at all key points, reflecting the intensity of the whole-body exercise.

[0184] ,

[0185] Among them, weight Different settings can be made according to different movement types, with the trunk joints usually given higher weight.

[0186] Step 5: Extracting the trajectory of the rope's motion;

[0187] In fancy rope skipping, the trajectory of the rope is key information for determining the type and quality of the movement. This invention uses the following method to extract the characteristics of the rope's motion.

[0188] (1) Rope body region segmentation

[0189] Preprocessed image frames In this process, color features or lightweight semantic segmentation networks are used to extract the rope region. Considering that jump ropes are typically high-contrast colors, the HSV color space thresholding method can be used for segmentation. Let the HSV value of the image pixel be... Rope mask Determined by the following formula:

[0190] ,

[0191] in The threshold parameters are preset based on the rope color.

[0192] (2) Positioning of the end point of the rope

[0193] Morphological refinement of the rope mask is performed to extract the rope's skeleton line. Among the two ends of the skeleton line, the rope end point is determined based on its distance from the wrist key point: the end point closer to the wrist key point is the handle end (usually obscured by the hand and difficult to detect stably), and the end point farther away is the free end (rope end).

[0194] Let the set of skeleton line endpoints be The coordinates of the key points on the wrist are and , the end point of the rope body Selected as:

[0195] ,

[0196] That is, select the skeletal end closest to any wrist as the end of the rope.

[0197] (3) Extraction of rope motion features

[0198] Coordinates of the end of the rope in consecutive frames Extract the following features:

[0199] Rope swing direction: Defined as the end of the rope in the horizontal plane ( The direction angle of motion (in a plane):

[0200] ,

[0201] Direction switching event: When the change in direction angle between two adjacent frames exceeds a preset threshold. When the angle is 90°, mark the current frame as the direction switching frame:

[0202] ,

[0203] Rope trajectory curvature: The local curvature of the rope's end trajectory fitted within a sliding window, reflecting the smoothness of the rope's motion. For three points within the window... The curvature is approximately:

[0204] ;

[0205] Step 6: Spatiotemporal feature modeling and action sequence recognition;

[0206] After acquiring the skeletal keypoint sequence, kinematic features, and rope motion features, a spatiotemporal feature representation is constructed and action recognition is performed.

[0207] (1) Construction of skeletal spatiotemporal map

[0208] Modeling the human skeletal sequence as a spatiotemporal graph The set of nodes Indicates the first The first frame A joint. Edge set. It consists of two parts:

[0209] Space edge Based on the natural connections of the human skeleton, adjacent joints are connected within the same frame, such as shoulder-elbow, elbow-wrist, hip-knee, etc.

[0210] Time edge : Connecting corresponding nodes of the same joint in adjacent frames, i.e. and The edges between them.

[0211] Each node The feature vector is composed of the joint's coordinates, velocity, acceleration, and the corresponding rope state.

[0212] (2) Spatiotemporal Graph Convolutional Network

[0213] An improved spatiotemporal graph convolutional network (ST-GCN) is used to extract features from the skeletal spatiotemporal graph. For nodes... Its neighborhood includes the spatial neighborhood. and time neighborhood The graph convolution operation is defined as follows:

[0214] ,

[0215] in and These are the input and output features, respectively. and The learnable spatial and temporal convolution weight matrix. and This is the normalization factor.

[0216] To adapt to the characteristics of short and fast-paced rhythmic sequences in fancy rope skipping, a multi-scale temporal convolution module is introduced into the network. Parallel processing with dilated convolutions of varying dilation rates is employed to capture movement patterns across different time spans.

[0217] ,

[0218] in For the set of expansion rates, For the first The expansion step size of each branch, This represents the number of branches.

[0219] (3) Integration of rope features

[0220] Rope motion features (direction switching indicators, curvature sequences) are fused with temporal features as auxiliary modalities. A cross-modal attention mechanism is employed, using skeletal features as queries and rope features as keys and values, to compute the enhancement of the skeletal representation by rope information.

[0221] ,

[0222] in , , , and These are sequences of skeletal features and rope features, respectively.

[0223] The fused features are mapped to an action category probability distribution through a fully connected layer.

[0224] (4) Temporal action segmentation and recognition

[0225] Sliding window processing is applied to the video stream, with a window length of [missing information]. The duration of the action is set according to the expected duration (e.g., the number of frames corresponding to 2 to 3 seconds). Within each window, the model outputs an action category probability vector. ,in This represents the number of action categories.

[0226] To accurately locate the start and end boundaries of the action, a dynamic time warping algorithm is used to align the predicted action sequence with a standard action template. Let the predicted sequence be... The template sequence is Both are skeletal keypoint sequences (normalized). A distance matrix is ​​defined. ,in DTW solves for the optimal alignment path using dynamic programming, and the cumulative distance matrix is ​​calculated as follows:

[0227] ,

[0228] The alignment point corresponding to the optimal path is the start and end boundary of the action.

[0229] Step 7: Action quality assessment and intelligent referee scoring;

[0230] After motion recognition is completed, the process moves to the intelligent referee scoring stage. The scoring system covers difficulty assessment, completion quality evaluation, violation detection, and comprehensive scoring.

[0231] (1) A database of standard movements for fancy rope skipping was established.

[0232] At the level of constructing the movement system, the work of sorting out the domestic and foreign fancy rope skipping movement database has been completed, and the basic movements (level one movements such as single rope skipping) and difficulty combinations (level two movements such as double-handed cross-leg straight rope skipping and cross-leg straight rope skipping behind the neck) have been systematically included. A standard movement database has been established by combining the standard movement library of the International Jump Rope Federation (IJRU) with the sports characteristics of champion athletes.

[0233] Table 1 Database of Standard Movements for Fancy Rope Skipping

[0234]

[0235] (2) Action difficulty level mapping

[0236] Establish a mapping table between action labels and difficulty levels. The difficulty level of basic actions (such as single rocking) is denoted as... Other moves are assigned a relative difficulty level based on competition rules and expert experience. :

[0237] Table 2 Difficulty coefficient of fancy rope skipping

[0238]

[0239] Total difficulty score of a performance The sum of the difficulty coefficients of all identified valid actions:

[0240] , in For the first The number of times a type of action is effectively completed.

[0241] (2) Complete the quality assessment

[0242] For each identified action, the quality is quantified by comparing the deviation between the actual skeletal trajectory and the standard action template.

[0243] Set action The standard template is a skeletal keypoint sequence. The actual detected sequence is (The lengths are consistent after DTW alignment). Define the joint. At any moment Deviation:

[0244] ,

[0245] Considering the varying importance of different joints during movement, joint weights are introduced. (Determined by the type of movement; for example, in a cross jump, the wrist has a greater weight than the ankle). The normalized overall bias of the movement is:

[0246] ,

[0247] The deviation is converted into a quality score using a preset mapping function. :

[0248] ,

[0249] in The sensitivity coefficient controls the steepness of the score's response to bias. The adjustment can be dynamically made based on the athlete's level—a larger adjustment is adopted for high-level athletes. To enhance differentiation, a smaller [size / weight] is used for beginners. To maintain tolerance in the scoring.

[0250] Overall performance quality score The weighted average of the quality scores for all movements:

[0251] ,

[0252] in The difficulty coefficient of the move is used to give greater weight to the quality of high-difficulty moves in the total score.

[0253] (3) Detection of violations

[0254] Establish a violation rule database to automatically identify and deduct points. Typical violation types include:

[0255] Rope stepping interruption: Multiple consecutive frames (e.g., ≥5 frames) were detected where the foot key points overlapped with the rope mask area, and the subsequent jump cycle was abnormally prolonged.

[0256] Hand position too low: The height of the wrist key point is consistently lower than the height of the hip by a certain proportion (such as more than 30% of the duration of the movement).

[0257] Missing Rope: By comparing the rope direction switching sequence with the jumping cycle, it can be determined whether there are situations where the rope has not completely wrapped around the body.

[0258] Excessive body sway: The standard deviation of lateral displacement at key trunk points (center of shoulder and hip) exceeds the threshold.

[0259] For each violation, points will be deducted according to the severity. (Violation point deduction) This is the sum of the points deducted for each violation.

[0260] (4) Calculation of comprehensive score

[0261] The final score is calculated by combining the difficulty score, the quality score, and the penalty points for violations:

[0262] ,

[0263] in This is the total difficulty score (the raw, unscaled value). For normalized quality scores (e.g., on a percentage scale). Points will be deducted for violations.

[0264] For easier and more intuitive presentation, you can Map to a preset scoring range (e.g., 0 to 100 points) or a rating system (A, B, C, D).

[0265] Step 8: System Deployment and Real-time Inference;

[0266] Deploy the trained model to a real-world application environment. Considering real-time requirements, the following optimizations are performed:

[0267] (1) Model quantization and acceleration

[0268] TensorRT or ONNX Runtime is used to accelerate model inference. INT8 quantization is performed on the YOLOv8-pose part, significantly improving inference speed while maintaining controllable accuracy loss. The quantization process converts floating-point weights... Mapped to integer :

[0269] ,

[0270] in To quantize the step size, This is the zero-point offset. The inverse quantization formula is: .

[0271] (2) Multithreaded pipeline design

[0272] Video acquisition, preprocessing, model inference, postprocessing, and scoring calculation are all placed in independent threads, and data is transferred through a shared memory queue, making full use of multi-core CPU / GPU resources and reducing end-to-end latency.

[0273] (3) Visual output

[0274] It displays real-time skeletal point connections, current action label, score, remaining time, and violation alerts. The visualization interface is built based on the OpenCV or Qt framework and overlays original video frames.

[0275] Output result:

[0276] The final output of this invention includes two parts: real-time visual feedback and a post-competition comprehensive scoring report. During system operation, the interface displays in real-time the currently identified action tags and confidence levels, the superimposed image of connecting key points of the human skeleton and the rope's movement trajectory. The right panel dynamically updates the comprehensive score, difficulty score, completion quality score, violation deductions, and the cumulative number of each action. The bottom timeline uses colored bars to mark the temporal distribution of identified actions and lists action details and violation events. After the competition or training session, the system automatically generates a complete scoring report, including athlete information, total score and rating, scoring trend charts for each dimension, action completion statistics, violation record list, action timeline visualization chart, and multi-dimensional ability radar chart. It also supports exporting PDF documents or structured JSON data for use by the event management system, thus achieving end-to-end automated processing from video input to professional judge scoring output.

Claims

1. A method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching, characterized in that, Includes the following steps: Step 1: Multi-view video data acquisition and synchronization; Simultaneous video capture of synchronized rope skipping athletes is conducted using a camera system with at least two viewing angles, with two video streams respectively. and The two cameras are synchronized via hardware triggering or software synchronization based on audio feature alignment; the video capture frame rate is set to... FPS; The captured motion types cover single jump, double jump, triple jump, cross jump, choreographed jump, side swing cross and combination movements; Step 2: Video preprocessing and frame extraction; The acquired video stream and Image frames are extracted at fixed frame intervals to form an image sequence; The following preprocessing operations are performed sequentially, among which Indicates the viewpoint number. Indicates the frame number corresponding to the timestamp: Scale the image to a uniform width and height. Pixels; Gaussian filtering or median filtering is used to remove random noise from the image; the preprocessed image sequence is denoted as... ; Step 3: Extraction of skeletal key points based on the improved YOLOv8-pose; For each frame of the preprocessed image An improved YOLOv8-pose model was used for human detection and skeletal key point extraction. YOLOv8-pose extends the pose estimation branch on the basis of the YOLOv8 object detection framework, and outputs the human bounding box and the coordinates of 17 skeletal key points simultaneously in a single forward propagation. For multi-view videos, detection is performed for each view to obtain a keypoint sequence. and ; The keypoint detection results from two perspectives at the same time are fused; Step 4: Post-processing of skeletal keypoint sequences and extraction of kinematic features; The fused skeletal keypoint sequence Post-processing and kinematic feature extraction are performed. ; Step 5: Extracting the trajectory of the rope's motion; Step 6: Spatiotemporal feature modeling and action sequence recognition; After acquiring the skeletal key point sequence, kinematic features, and rope motion features, a spatiotemporal feature representation is constructed and action recognition is performed. Step 7: Action quality assessment and intelligent referee scoring; After motion recognition is completed, the process enters the intelligent referee scoring stage; the scoring system covers difficulty assessment, completion quality evaluation, violation detection, and comprehensive scoring. Step 8: System Deployment and Real-time Inference; Deploy the trained model to a real-world application environment.

2. The method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching according to claim 1, characterized in that, In the second step, scaling is performed using a bilinear interpolation algorithm.

3. The method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching according to claim 1, characterized in that, In the third step, a CA attention mechanism is embedded in the C2f module of the YOLOv8 backbone network; the channel attention is decomposed into two one-dimensional feature encoding processes, which aggregate features along the horizontal and vertical directions respectively to capture precise location information.

4. The method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching according to claim 1, characterized in that, The specific method for the fourth step is as follows: (1) Temporal smoothing filter A Kalman filter is used to smooth the two-dimensional trajectory of each keypoint. Through prediction-update iteration, the smoothed keypoint coordinate sequence is obtained. ; (2) Interpolation of missing key points For keypoints marked as missing, cubic spline interpolation is used to complete them; let the coordinates of the known keypoints before and after the missing segment be... and Construct a piecewise cubic polynomial that satisfies the boundary second-order continuity for interpolation; (3) Calculation of kinematic characteristics Based on the smoothed keypoint sequence, kinematic features are calculated for subsequent action recognition and quality assessment.

5. The method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching according to claim 1, characterized in that, The specific method for the fifth step is as follows: (1) Rope body region segmentation Preprocessed image frames In this process, the rope region is extracted using color features or a lightweight semantic segmentation network; an HSV color space thresholding method is employed; and the HSV value of each image pixel is set to... Rope mask Determined by the following formula: ， in The threshold parameters are preset based on the rope color; (2) Positioning of the end point of the rope Morphological refinement of the rope mask was performed to extract the rope's skeleton lines; Among the two ends of the skeleton line, the end point of the rope is determined according to the distance relationship with the wrist key point: the end point closer to the wrist key point is the handle end, and the end point farther away is the free end, i.e. the end of the rope. Let the set of skeleton line endpoints be The coordinates of the key points on the wrist are and , the end point of the rope body Selected as: ， That is, select the skeletal end closest to any wrist as the end of the rope; (3) Extraction of rope motion features Coordinates of the end of the rope in consecutive frames Extract the following features: Rope swing direction: Defined as the end of the rope in the horizontal plane ( The direction angle of motion (in a plane): ， Direction switching event: When the change in direction angle between two adjacent frames exceeds a preset threshold. When the angle is 90°, mark the current frame as the direction switching frame: ， Rope trajectory curvature: The local curvature of the rope's end trajectory fitted within a sliding window, reflecting the smoothness of the rope's motion; for three points within the window... The curvature is approximately: 。 6. The method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching according to claim 1, characterized in that, The specific method for step six is ​​as follows: (1) Construction of skeletal spatiotemporal map Modeling the human skeletal sequence as a spatiotemporal graph ; where the node set Indicates the first The first frame One joint; edge set It consists of two parts: Space edge Based on the natural connection relationships of the human skeleton, connect adjacent joints within the same frame; Time edge : Connecting corresponding nodes of the same joint in adjacent frames, i.e. and The edge between; Each node The feature vector is composed of the joint's coordinates, velocity, acceleration, and the corresponding rope state. (2) Spatiotemporal Graph Convolutional Network An improved spatiotemporal graph convolutional network is used to extract features from the skeletal spatiotemporal graph; for nodes... Its neighborhood includes the spatial neighborhood. and time neighborhood The graph convolution operation is defined as follows: ， in and These are the input and output features, respectively. and The learnable spatial and temporal convolution weight matrix. and Normalization factor; A multi-scale temporal convolution module is introduced into the network, employing dilated convolutions with different dilation rates in parallel processing to capture action patterns across different time spans: ， in For the set of expansion rates, For the first The expansion step size of each branch, Number of branches; (3) Integration of rope features The rope motion characteristics are used as auxiliary modes and fused with temporal characteristics; A cross-modal attention mechanism is employed, using skeletal features as the query and rope features as the key and value, to compute the enhancement of the skeletal representation by rope information: ， in , , , and These are sequences of skeletal features and rope features, respectively. The fused features are mapped to an action category probability distribution through a fully connected layer; (4) Temporal action segmentation and recognition Sliding window processing is applied to the video stream, with a window length of [missing information]. The duration is set according to the expected action duration; within each window, the model outputs an action category probability vector. ,in Number of action categories; To accurately locate the start and end boundaries of the action, a dynamic time warping algorithm is used to align the predicted action sequence with a standard action template; let the predicted sequence be... The template sequence is Both are sequences of skeletal keypoints; a distance matrix is ​​defined. ,in DTW solves for the optimal alignment path using dynamic programming, and the cumulative distance matrix is ​​calculated as follows: ， The alignment point corresponding to the optimal path is the start and end boundary of the action.

7. The method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching according to claim 1, characterized in that, The seventh step specifically includes: (1) A database of standard movements for fancy rope skipping was established. (2) Action difficulty level mapping Establish a mapping table between action labels and difficulty levels; the difficulty level of basic actions is denoted as... Other moves are assigned a relative difficulty level based on competition rules and expert experience. : Total difficulty score of a performance The sum of the difficulty coefficients of all identified valid actions: (2) Complete the quality assessment For each identified action, the quality is quantified by comparing the deviation between the actual skeletal trajectory and the standard action template. Set action The standard template is a skeletal keypoint sequence. The actual detected corresponding sequence is Define joints At any moment Deviation: ， Considering the varying importance of different joints during movement, joint weights are introduced. The normalized overall bias of the action is: ， The deviation is converted into a quality score using a preset mapping function. : ， in The sensitivity coefficient controls the steepness of the score's response to bias. Overall performance quality score The weighted average of the quality scores for all movements: ， in The difficulty coefficient of the move is used to give greater weight to the quality of high-difficulty moves in the total score. (3) Detection of violations Establish a rule violation database to automatically identify and deduct points for violations; deduct points for violations. This is the sum of the points deducted for each violation. (4) Calculation of comprehensive score The final score is calculated by combining the difficulty score, the quality score, and the penalty points for violations: ， in This is the total difficulty score, the raw, unscaled value. The normalized quality score; Points will be deducted for violations.

8. The method for recognizing and intelligently judging fancy rope skipping movements based on visual spatiotemporal feature matching according to claim 1, characterized in that, Step 8 also includes the following optimizations: (1) Model quantization and acceleration Accelerate model inference using TensorRT or ONNX Runtime; INT8 quantization is performed on the YOLOv8-pose part, which significantly improves inference speed while keeping the accuracy loss controllable; the quantization process reduces floating-point weights. Mapped to integer : ， in To quantize the step size, Zero-point offset; The inverse quantization formula is ; (2) Multithreaded pipeline design Video acquisition, preprocessing, model inference, postprocessing and scoring calculation are placed in separate threads, and data is transferred through a shared memory queue to make full use of multi-core CPU or GPU resources and reduce end-to-end latency. (3) Visual output It displays real-time skeletal point connections, current action labels, scores, remaining time, and violation warnings; the visualization interface is built based on OpenCV or Qt frameworks and overlays original video frames.

Images