An outdoor skiing intelligent motion analysis method and system based on lightweight OpenPose and space-time tube, and a storage medium

Abstract

The application discloses an outdoor skiing intelligent action analysis method and system based on light OpenPose and a space-time tube and a storage medium, and belongs to the technical field of intelligent ice and snow sports. The method adopts a CSP-Darknet53 main network to improve the OpenPose in a light manner, thereby improving the precision and robustness of human key point detection in a complex environment; meanwhile, an action quality evaluation model based on a space-time tube self-attention mechanism (TSA-Net) is introduced, a TSA module is used for space-time feature aggregation, a single target tracking method is used to associate continuous frame target information, a three-dimensional space-time tube is constructed by aggregating time sequence coordinates, and self-attention feature calculation is limited from global dense calculation to tube internal sparse calculation; after being spliced and fused with the key point coordinates and joint angle output by the modified OpenPose network, the action quality score is output by a multilayer perceptron; the method effectively suppresses background noise, focuses on the movement characteristics of athletes, and realizes accurate real-time action scoring.

Description

Technical Field

[0001] This invention belongs to the field of intelligent technology for ice and snow sports, specifically relating to an intelligent motion analysis method, system, and storage medium for outdoor skiing based on lightweight OpenPose and spatiotemporal management. Background Technology

[0002] In recent years, winter sports have received increasing attention, and scholars have dedicated themselves to research on sports training, leading to the continuous development of technologies and products for skiing posture analysis. This research is mainly divided into two categories based on data sources: one is wearable devices, such as wristwatches, vests, and helmets, which utilize physiological sensors to describe the wearer's body posture, thereby detecting the athlete's posture and judging the standard of their movements. Physiological sensors need to be in close contact with the human body for data collection, which can easily affect the accurate representation of the subject's skiing technique. Furthermore, the sensors are expensive and easily affected by the data acquisition device and environment, making this method difficult to collect data. The other category is based on vision and machine learning to detect skier movement models. This method uses cameras to capture training footage of athletes, extracts the human skeleton in motion using algorithms, compares it with the human skeleton in standard movement, and records the movements in real time. Visual sensor cameras do not need to be close to the subject to collect data, offering advantages such as non-contact, low cost, and low environmental impact. Camera-based posture detection can provide skiers with a better training experience and has greater practicality, but existing camera-based posture detection methods suffer from high computational load, susceptibility to complex background interference, and poor real-time performance. Therefore, this invention designs a vision-based ski training posture analysis system, which is more conducive to the evaluation requirements of skiing actions in real-world scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide an intelligent motion analysis method for outdoor skiing based on lightweight OpenPose and spatiotemporal management, which solves the problems of large computational load, susceptibility to interference from complex backgrounds, and poor real-time performance in existing technologies.

[0004] This invention provides an intelligent motion analysis method for outdoor skiing based on lightweight OpenPose and spatiotemporal management, comprising the following steps:

[0005] Acquire multi-angle training video data and extract video frame images; manually annotate the coordinates of key points and joint angles of the human body in each frame image, and score the motion quality to construct a dataset;

[0006] A modified OpenPose network was constructed. The Darknet53 network with CSP modules introduced at the head end was used as the backbone network to extract features. Then, the key point heatmap and limb affinity field features were iteratively optimized through a multi-stage cascaded double-branch network to output the coordinates of human key points. Finally, the joint angle calculation branch embedded at the end of the modified OpenPose network was used to output the angles of each joint.

[0007] A TSA-Net network is constructed; the TSA module is used to aggregate spatiotemporal features, and the target information of continuous frames is associated through the single target tracking method. The temporal coordinates are aggregated to construct a three-dimensional spatiotemporal tube, and the self-attention feature calculation is restricted from global dense calculation to sparse calculation within the tube. After being spliced ​​and fused with the key point coordinates and joint angles output by the modified OpenPose network, the motion quality score is output through a multilayer perceptron.

[0008] The training images to be detected are sequentially input into the trained modified OpenPose network and TSA-Net network, and the action quality scores are output.

[0009] Furthermore, the key points include the head, chest, waist, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left foot, and right foot; the joint angles include the hip joint angle, knee joint angle, shoulder joint angle, and elbow joint angle.

[0010] Furthermore, the image is input into the backbone network CSP-Darknet53. After feature extraction through basic layer convolution, it is input into the CSP module, which splits it into two branches along the channel dimension for separate processing. One branch goes directly to the output port, and the other is sent to the dense block for deep feature extraction. After fusing shallow detail features and deep semantic features, the image is output after subsequent convolution to restore the number of channels. Then, it is processed by the Darknet53 network to output multi-scale hierarchical features.

[0011] Furthermore, the multi-stage cascaded dual-branch network obtains candidate key point coordinates through heatmap peak retrieval and coordinate mapping processing, and completes human joint bipartite map matching and grouping by combining limb affinity field features, outputting two-dimensional key point coordinates.

[0012] Furthermore, during the training phase, the backbone network parameters are first fixed, and only the multi-stage dual-branch iterative CNN module and the joint angle branch are trained; then the modified OpenPose network is jointly fine-tuned.

[0013] The loss function for the training phase is:

[0014]

[0015] in, Loss for the two-dimensional probabilistic heatmap of key points; This results in the loss of some affinity domains. This is due to joint angle loss; , , These are the weighting coefficients.

[0016] Furthermore, the spatiotemporal feature aggregation performed by the TSA module specifically involves:

[0017] Step 1.1: Generate the bounding box of the skier in each frame of the video sequence using the target tracking algorithm, arrange the bounding boxes of all frames in chronological order, and perform Kalman filtering to smooth the sequence; stack the smoothed T bounding boxes on the time axis to form a three-dimensional spatiotemporal tube.

[0018] Step 1.2: Using the I3D network as the backbone network, extract image features frame by frame through 3D convolutional kernels; map the spatiotemporal tube region onto the feature map scale, downsample the outer bounding box to obtain the spatial region on the corresponding feature map; perform self-attention feature interaction on the features within the spatial region;

[0019] The formula for calculating the coordinate downsampling of the bounding rectangle onto the feature map is as follows:

[0020]

[0021]

[0022] in, For the first The minimum column index on the feature map that the frame's bounding rectangle is mapped to. For the first The maximum column index on the feature map that the frame's bounding rectangle maps to; For the first The minimum row index on the feature map that the frame's bounding rectangle is mapped to; For the first The maximum row index on the feature map that the frame's bounding rectangle is mapped to; and For the first The horizontal and vertical coordinates of the center of the frame bounding rectangle in the original input frame; and For the first The width and height of the frame bounding rectangle in the original input frame; This represents the sampling multiple.

[0023] Furthermore, the preprocessed keypoint coordinates and joint angles are concatenated, and then concatenated with the spatiotemporal features obtained by global average pooling to obtain fused features; the fused features are input into the multilayer perceptron to output the score of each action.

[0024] Furthermore, the loss function of the multilayer perceptron is:

[0025]

[0026] in, , These are the predicted and actual score distributions, respectively. The number of discretizations for the score levels; Indicates the first Each score level.

[0027] An outdoor skiing intelligent motion analysis system based on lightweight OpenPose and spatiotemporal management includes:

[0028] Data acquisition and preprocessing module: Completes multi-angle training video data acquisition, frame image extraction and annotation of key points and joint angles of the human body, as well as motion quality scoring; constructs dataset and performs unified preprocessing operations on images and labeled data within the dataset;

[0029] Lightweight human keypoint detection module: The Darknet53 network with CSP module introduced at the head end is used as the backbone network to extract features. Then, the keypoint heatmap and limb affinity field features are iteratively optimized through a multi-stage cascaded double-branch network to output the coordinates of human keypoints. Then, the joint angle calculation branch embedded at the end of the modified OpenPose network is used to output the angles of each joint.

[0030] The motion quality assessment module of the spatiotemporal tube: The TSA module is used to aggregate spatiotemporal features. The target information of continuous frames is associated through the single target tracking method. The temporal coordinates are aggregated to construct a three-dimensional spatiotemporal tube. The self-attention feature calculation is restricted from global dense calculation to sparse calculation within the tube. After being spliced ​​and fused with the key point coordinates and joint angles output by the modified OpenPose network, the motion quality score is output through the multilayer perceptron.

[0031] Real-time feedback module: Provides real-time visualization results of the human skeleton and motion quality scores.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube as described above.

[0033] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube as described above.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention significantly improves the accuracy of human keypoint detection under complex backgrounds, varying lighting conditions, and partial occlusion by constructing a posture dataset specifically for outdoor skiing scenarios and using a lightweight improvement to OpenPose with the CSP-Darknet53 backbone network. Simultaneously, the TSA-Net module introduces a spatiotemporal management mechanism to effectively suppress background noise and interference from athlete occlusion, enabling the motion quality assessment model to accurately focus on the athlete's own motion characteristics, thus providing more reliable scoring results. After TensorRT optimization, the system's model can run at speeds exceeding 30 FPS on the Jetson Nano embedded platform with end-to-end latency below 50ms. The system supports 17 keypoint detections and joint angle calculations, and integrates multimodal feedback from vision, speech, and cloud storage, providing a complete solution for non-contact, low-cost digitization of ski training. This invention combines high precision, real-time performance, low power consumption, and wide applicability, and can be applied to technical optimization training for professional athletes, especially suitable for general ski instruction, ski club skills assessment, and winter sports education. Its lightweight and real-time characteristics provide a feasible technical path for the popularization of intelligent equipment for ice and snow sports, and help promote the transformation and upgrading of the ice and snow economy from "seasonal experience" to "technology-based service". Attached Figure Description

[0036] Figure 1 This is a flowchart of the intelligent motion analysis method for outdoor skiing based on lightweight OpenPose and spatiotemporal tube of the present invention;

[0037] Figure 2 This is the overall design architecture diagram of the outdoor skiing intelligent motion analysis system of the present invention;

[0038] Figure 3 This is a flowchart of the action scoring network architecture of the method of the present invention. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings. The embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0040] An outdoor skiing intelligent motion analysis system based on lightweight OpenPose and spatiotemporal tube includes: a data acquisition and preprocessing module, a lightweight human key point detection module, a motion quality assessment module based on spatiotemporal tube, and a real-time feedback module.

[0041] The data acquisition and preprocessing module is used to build a ski-specific dataset.

[0042] In an outdoor ski resort, training videos of skiers were captured using multi-angle cameras. Keyframe extraction algorithms were used to extract image frames containing clear skiing movements from the videos. Images were manually screened to remove blurry, heavily occluded, or repetitive images. Data augmentation techniques were used to augment the selected images, forming an original image set. Annotation tools were used to label key points of the human body in the image set and calculate joint angles. At the same time, each movement image was labeled with a quality score according to skiing technique scoring rules (such as snow tricks, turns, and jumps). The labeled dataset was then randomly divided into training, validation, and test sets according to a preset ratio.

[0043] A lightweight human key point detection module is used to detect the coordinates of human key points and joint angles;

[0044] A human keypoint detection network based on an improved OpenPose was constructed. The VGG19 backbone network in the original OpenPose network was replaced with a Darknet53 network with a cross-stage local network (CSPNet) structure to reduce network redundancy and computational load. The keypoint structure of the OpenPose network was optimized and adjusted to a 17-keypoint model adapted to skiing motion analysis, and joint angle information was added as output. The improved lightweight OpenPose network was trained using the dataset constructed in step 1 to obtain a pose estimation model that can extract the skier's human skeleton and joint angles in real time and accurately.

[0045] The motion quality assessment module of the spacetime tube is used to score the motion of images.

[0046] A Tube Self-Attention Network (TSA-Net) combining target tracking and self-attention mechanisms is constructed. This network first uses a target tracking algorithm (such as SiamMask) to generate a bounding box for the skier in each frame of the video sequence, thus determining the "spatiotemporal tube" (ST-Tube) region where the athlete is located. Then, an I3D network is used as the backbone to extract video features, but sparse self-attention feature interactions are only performed within the determined "spatiotemporal tube" region. In this way, the network can ignore complex background noise and focus on the athlete's own motion features, while significantly reducing computational complexity. The human skeleton sequence, joint angles, and features output by I3D extracted in step 2 are used as input to a multilayer perceptron (MLP), and the network outputs a quality score for the current motion.

[0047] The real-time feedback module is used to provide skiers with real-time feedback on the visualization of the human skeleton and instant movement scores via a display screen or voice device.

[0048] An intelligent motion analysis system for outdoor skiing includes: a data acquisition and preprocessing module, a lightweight human key point detection module, a motion quality assessment module based on spatiotemporal management, and a real-time feedback module; the specific steps are as follows:

[0049] Step 1: Data collection and preprocessing to build a ski-specific dataset;

[0050] On-site at an outdoor ski resort, training videos of skiers were captured using multi-angle cameras. Keyframe extraction algorithms were used to extract image frames containing clear skiing movements from the videos. Images were manually screened to remove blurry, heavily occluded, or repetitive images. Data augmentation techniques were used to augment the selected images, forming an original image set. Annotation tools were used to label key points of the human body in the image set, and each movement image was labeled with a quality score according to skiing technique scoring rules (such as snow tricks, turns, and jumps). The labeled dataset was then randomly divided into training, validation, and test sets according to a preset ratio.

[0051] Step 2: Lightweight human body key point detection;

[0052] A human keypoint detection network based on an improved OpenPose was constructed. The VGG19 backbone network in the original OpenPose network was replaced with a Darknet53 network with a cross-stage local network (CSPNet) structure to reduce network redundancy and computational load. The keypoint structure of the OpenPose network was optimized and adjusted to a 17-keypoint model adapted to skiing motion analysis, and joint angle information was added as output. The improved lightweight OpenPose network was trained using the dataset constructed in step 1 to obtain a pose estimation model that can extract the skier's human skeleton and joint angles in real time and accurately.

[0053] Step 3: Motion quality assessment based on the spacetime tube;

[0054] A Tube Self-Attention Network (TSA-Net) combining target tracking and self-attention mechanisms is constructed. This network first uses a target tracking algorithm (such as SiamMask) to generate bounding boxes for the skier in each frame of the video sequence, thus determining the "spatiotemporal tube" (ST-Tube) region where the athlete is located. Then, an I3D network is used as the backbone to extract video features, but sparse self-attention feature interactions are only performed within the determined "spatiotemporal tube" region. In this way, the network can ignore complex background noise and focus on the athlete's own motion features, while significantly reducing computational complexity. The human skeleton sequence extracted in step 2 and the original video frames are used as input to the TSA-Net, and the network outputs a quality score for the current action.

[0055] Step 4: Model Deployment and Real-time Feedback;

[0056] The trained lightweight pose estimation model and motion quality assessment model are compressed and converted into their formats, and then deployed to an NVIDIA Jetson Nano embedded development board. Cameras and Jetson Nano devices with the deployed models are configured at an outdoor ski resort. Real-time video of skiers is captured, and the deployed models are used for real-time inference to obtain human skeleton visualization results and instant motion scores. Real-time feedback is then provided to skiers through a display screen or voice device.

[0057] An intelligent motion analysis method for outdoor skiing based on lightweight OpenPose and spatiotemporal management includes the following steps:

[0058] Step 1: Data collection and preprocessing to build a ski-specific dataset;

[0059] Step 1.1: Multi-source data acquisition and integration;

[0060] First, multiple high-definition cameras were deployed at an outdoor ski resort to simultaneously capture training videos of ski enthusiasts and professional athletes from different angles (e.g., front, side, top view) and distances (close-up, long view). This ensured coverage of various skiing techniques (e.g., downhill skiing, snowplow turns, parallel turns, snow tricks, jumps, etc.) and scenes under different lighting conditions (sunny, cloudy, backlight) and complex backgrounds (snow slopes, forests, crowds). Simultaneously, publicly available human pose datasets (e.g., Human3.6M) and dance motion datasets (e.g., AIST++) related to movement postures were incorporated as supplementary training data to enhance the model's generalization ability. All captured videos were saved at high frame rates (≥30fps) and high resolutions (≥1080p), and categorized by date, location, and movement type.

[0061] Step 1.2: Keyframe extraction and data filtering;

[0062] Using a Python script to call the OpenCV library, keyframes were extracted from the acquired raw video. By calculating inter-frame differences or using a pre-trained motion classifier, video frames containing obvious skiing movements and complete human postures were automatically selected and converted into JPEG still images. Subsequently, professionals manually screened the extracted images, removing those with low quality due to motion blur, severe occlusion, overexposure or underexposure, and those with excessively repetitive movements. To ensure data diversity, samples from athletes of different body types, skiing equipment, and movement ranges were retained.

[0063] Step 1.3: Data augmentation and annotation;

[0064] Data augmentation was performed on the selected image set using methods such as random rotation (±15°), horizontal flipping, random cropping, color jittering (brightness, contrast, and saturation adjustment), and Gaussian noise addition to expand the original image set to 3-5 times its original size. This increased data diversity and prevented model overfitting. The augmented images were manually annotated using the Labelme annotation tool: First, precise point annotations were performed on the human joints according to the 17 keypoints defined in the COCO dataset (including head, chest, waist, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left foot, and right foot). Second, based on the FIS's "Freestyle Skiing Moguls Technical Scoring Guidelines," the quality of each action image was scored by experts (e.g., 0-100 points), and the scores were used as labels for the action quality assessment task. After annotation, all annotation information was compiled into a JSON format label file. Finally, the entire dataset was randomly divided into training, validation, and test sets in a 7:2:1 ratio to ensure that the data distribution of the three subsets was consistent, laying the foundation for subsequent model training and performance evaluation.

[0065] Step 2: Lightweight human body key point and joint angle detection

[0066] Step 2.1: Improved backbone network design;

[0067] To reduce the computational complexity and parameter count of the VGG19 backbone in the original OpenPose network, this invention employs Darknet53 with a Cross-Stage Partial Network (CSPNet) structure as the new backbone. CSP-Darknet53 introduces a CSP module based on Darknet53, its core being feature splitting and cross-stage merging mechanisms. Specifically, at the entry point, the CSP module uses basic layer convolutions to evenly divide the input feature map into two parts along the channel dimension: one part is directly passed to the module exit without participating in any convolution calculations, thus preserving the original low-order features and reducing computational redundancy; the other part is fed into a dense block for deep feature extraction to capture high-order semantic information. The two paths are merged and fused at the module exit through cross-stage merging, and then output after subsequent convolutions to restore the number of channels. This allows gradients to flow through both paths simultaneously during backpropagation, resulting in richer gradient combinations, maintaining feature extraction capabilities while reducing network redundancy and improving inference speed. The specific replacement method is as follows: the input image (normalized to 368×368) is input into the CSP-Darknet53 network to extract multi-layer feature maps; the feature map output by the backbone network (46×46×1024) is used as the input to the subsequent OpenPose stage network. This improvement allows the backbone network to maintain its feature extraction capabilities while reducing the number of parameters by about 30% and the computational cost by about 25%, laying the foundation for subsequent deployment on edge devices.

[0068] Step 2.2: Key point detection and angle calculation branch design;

[0069] Following the backbone network, the original OpenPose multi-stage cascaded architecture is retained, including the first stage (Stage 1) and subsequent refinement stages (Stage 2~Stage 6). The core design idea of ​​this architecture is to gradually optimize the accuracy of keypoint localization and limb association through multi-stage iterative prediction. Specifically, the first stage takes the feature map output by the backbone network as input and generates an initial keypoint confidence map and a partial affinity domain through two parallel branches; the subsequent refinement stages concatenate the outputs of the two branches of the previous stage with the feature map of the backbone network and use them together as input. In addition, an intermediate supervision mechanism is introduced at the end of each stage. By calculating the loss between the current prediction and the true value and weighting it into the total loss, the gradient vanishing problem in deep networks is effectively prevented, guiding the network to continuously optimize in the correct direction during the iteration process. Each stage comprises two branches: Branch 1 predicts Part Confidence Maps, which generates two-dimensional probability heatmaps for each keypoint type, obtaining the pixel coordinates of 17 keypoints through peak localization; Branch 2 predicts Part Affinity Fields (PAFs), which encodes the orientation and position information of each limb connection in the form of a two-dimensional vector field, used to solve the keypoint matching problem in multi-target, occluded scenes. For skiing motion analysis, the number of keypoints was adjusted from the original 18 (COCO format) in OpenPose to 17 keypoints adapted to skiing postures (including head, chest, waist, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left foot, and right foot). The coordinates of candidate body keypoints in the two-dimensional probability heatmaps of each frame generated by the keypoint confidence map prediction branch are extracted. Combined with the corresponding frame limb vector fields generated by the partial affinity field prediction branch, the human joint bipartite map matching and grouping are completed, outputting the two-dimensional keypoint coordinates at each time step. Building upon the two parallel branches of OpenPose, this invention adds a joint angle calculation branch at the network's end. This branch takes the pixel coordinates of 17 keypoints output from branch 1 as input and calculates the main joint angles in real-time through vector dot product operations. These include the hip joint angle (angle between the thigh and torso), knee joint angle (angle between the thigh and calf), shoulder joint angle (angle between the upper arm and torso), and elbow joint angle (angle between the upper arm and forearm). This branch can be embedded as an independent module at the network's end or implemented through post-processing, providing refined pose features for subsequent motion quality assessment.

[0070] Step 2.3: Model training and optimization;

[0071] The improved lightweight OpenPose network was trained end-to-end using the skiing posture dataset constructed in step 1. This network employs a divide-and-conquer training scheme, with the training process divided into two stages: First, the backbone network parameters are fixed, and only the multi-stage cascade (Stage 1~Stage 6) and joint angle branches are used as trainable parameters, allowing the cascade to quickly converge to a reasonable posture estimation space. The training epochs are set to 60. Second, the entire network is jointly fine-tuned, the backbone network parameters are unfrozen, and the backbone and multi-stage cascade are jointly trained. The loss function consists of three parts: the mean squared error loss of the keypoint confidence map, the mean squared error loss of the PAF, and the mean squared error loss of the joint angle calculation branch (the ground truth angles are calculated from manually labeled keypoint coordinates). The training input consists of a skier image and the corresponding ground truth values ​​of the keypoint confidence map, PAF, and joint angles. The output consists of predicted heatmaps of 17 keypoints, the PAF vector field, and the calculated joint angles.

[0072]

[0073]

[0074]

[0075]

[0076] in, Loss for the two-dimensional probabilistic heatmap of key points; This results in the loss of some affinity domains. This is due to joint angle loss; , , These are the weighting coefficients for each loss term; and They are respectively No. Predicted and actual values ​​of the confidence plot for each key point Number of key points; and The first Predicted values ​​and actual values ​​of partial affinity domains for limb connections For the number of limb connections; and These are the predicted and actual values ​​of the angle of the i-th joint, respectively. The number of joint angles. The feature map space size.

[0077] The total loss function is a weighted sum of the three factors, with the weight coefficients determined experimentally. The Adam optimizer was used, with an initial learning rate of 1e-4, decaying every 10 epochs with a decay factor of 0.1. Data augmentation (random rotation, scaling, and flipping) was applied during training to improve model robustness. After training, the model was converted to TensorRT format for real-time inference (≥25 FPS) on a Jetson Nano, outputting a pose data stream containing coordinates of 17 keypoints and joint angle information.

[0078] Step 3: Performance evaluation based on tubular self-attention network and multilayer perceptron

[0079] Step 3.1: Generation of spatiotemporal control region based on target tracking;

[0080] SiamMask (Siamese Mask) is used as the target tracker, with the skier's position in the first frame of the video as the initial template. It tracks the skier in real-time in subsequent frames and generates accurate bounding boxes. SiamMask is an online tracking and segmentation algorithm based on Siamese networks, capable of simultaneously outputting rotated bounding boxes and target segmentation masks, and exhibiting strong robustness to non-rigid deformation and rapid motion. For an input video segment (containing T frames, each frame normalized to 224×224), the athlete's region detected by manual selection in the first frame is first extracted as a sample. Subsequently, SiamMask predicts the target's position in each subsequent frame, outputting the minimum bounding box. The bounding boxes of all frames are arranged chronologically, and the box sequence is smoothed using Kalman filtering to eliminate inter-frame jitter and tracking noise. The smoothed T bounding boxes are stacked on the time axis to form a three-dimensional "spatio-temporal tube" (ST-Tube). This spacetime tube precisely describes the range of spatial activity of an athlete throughout the entire movement sequence. Its spatial dimensions may vary over time, but it remains continuous in the temporal dimension. The formula for generating the spacetime tube can be expressed as:

[0081]

[0082] in, and For the first The x and y coordinates of the center of the bounding rectangle in the original input frame. and For the first The width and height of the frame bounding rectangle in the original input frame.

[0083] Step 3.2: Feature extraction based on I3D and tubular self-attention mechanism;

[0084] Inflated 3D ConvNets (I3D) are used as the backbone network to extract spatiotemporal features from the input video clip. I3D expands 2D convolutional kernels into 3D convolutional kernels, effectively capturing motion information. The input video clip has a size of T×224×224×3. After forward propagation through the I3D network, a new video clip with a size of T×224×224×3 is obtained. Feature map (in The length is the time dimension. , For spatial dimensions, (Number of channels). To map the spatiotemporal tube region to the feature map scale, the bounding rectangle of each frame generated in step 3.1 is downsampled. The downsampling formula is as follows:

[0085]

[0086]

[0087] in, For the first The minimum column index (left boundary) of the frame bounding rectangle is mapped to the feature map. For the first The maximum column index (right boundary) of the frame bounding rectangle mapped onto the feature map. For the first The minimum row index (upper boundary) of the frame bounding rectangle mapped onto the feature map. For the first The maximum row index (lower bound) of the frame bounding rectangle mapped onto the feature map; This represents the sampling multiple.

[0088] After downsampling, the spatial region on the corresponding feature map is obtained. Then, construct a 3D mask. In each frame t' (corresponding to the original frame t), the feature point is located at The internal mask value is 1 if the mask is set to 1, and 0 otherwise. Then, a self-attention mechanism is applied only to feature points where the mask value is 1. Specifically, all feature vectors within the mask are treated as a set of "tokens," and the self-attention between them is calculated to obtain the spatiotemporal features. ;

[0089]

[0090] in, , , Each token is obtained by linear projection from feature points within the mask. Through self-attention, each token can aggregate global information within the entire spatiotemporal tube, enhancing the discriminative power of the features. At the same time, since a large number of background feature points outside the tube are excluded, the computational complexity is significantly reduced. This module is called Tube Self-Attention (TSA), which realizes non-local modeling within local regions.

[0091] Step 3.3: Multimodal feature fusion and motion quality scoring;

[0092] This invention introduces a multimodal feature fusion strategy based on the TSA-Net pure vision architecture, constructing a motion quality assessment network that combines computational efficiency and scoring accuracy. Specifically, it boasts the following two core advantages: First, computational efficiency. This invention uses the TSA module for spatiotemporal feature aggregation and generates a 3D spatiotemporal tube through SiamMask, restricting self-attention computation from globally dense to sparse within the tube. Second, robustness of multimodal fusion. Pure vision methods rely solely on I3D appearance features, which are prone to degradation under interference such as lighting changes, snow reflections, and occlusion, and struggle to characterize fine-grained pose information such as joint angles and limb coordination. This invention, based on the TSA module, introduces the human skeleton sequence and joint angles extracted by the lightweight OpenPose in step 2, splicing and fusing appearance and structural information along the channel dimension, and outputting the motion score via an MLP network.

[0093] The specific implementation is as follows:

[0094] First, the skeleton sequence is preprocessed: the keypoint coordinates are normalized to the video frame size, and linear interpolation is used to align the sequence length to the same temporal dimension as the I3D features. The normalized keypoint coordinates and joint angles are concatenated into a feature vector. ( (For pose feature dimensions). Then, the spatiotemporal features are... Perform global average pooling to obtain the feature vector in the time dimension. .Will and By concatenating along the feature channel dimension, the fused features are obtained. .Will The input is fed into a multilayer perceptron (MLP) consisting of two fully connected layers, each followed by a ReLU activation function and Dropout (probability 0.5). The network outputs a fractional probability distribution. To reduce the subjective uncertainty of judges' scoring, the scoring is modeled as a Gaussian distribution. During training, the KL divergence loss from Uncertainty-Aware Score Distribution Learning (USDL) is used to measure the difference between the predicted score distribution and the Gaussian distribution corresponding to the true scores. The KL divergence loss function is:

[0095]

[0096]

[0097] in, , The score distributions are for prediction and actual (expert ratings), respectively. The number of discretizations for the score levels; Indicates the first Each score level.

[0098] The parameters of the entire TSA-Net are optimized through backpropagation, enabling it to accurately evaluate the quality of skiing actions. The TSA module ensures real-time inference capabilities for edge devices, while multimodal fusion improves scoring accuracy and anti-interference capabilities. This allows the network to maintain computational efficiency while effectively coping with the complex outdoor skiing environment, providing skiers with immediate and accurate feedback on the quality of their actions.

[0099] Step 4: Model Deployment and Real-time Feedback Step 4.1: Model Optimization and Embedded Platform Conversion

[0100] To achieve real-time inference in outdoor skiing scenarios, the trained lightweight OpenPose pose estimation model and TSA-Net motion quality assessment model need to be compressed and converted to adapt to the hardware resource limitations of the NVIDIA Jetson Nano embedded platform. First, the model trained under the PyTorch framework is exported to ONNX (Open Neural Network Exchange) format to achieve framework neutrality. Then, NVIDIA TensorRT is used to deeply optimize the ONNX model: FP16 precision quantization is adopted, converting model weights and activation values ​​from 32-bit floating-point numbers to 16-bit floating-point numbers, reducing model size and computational bandwidth with almost no loss of accuracy; layer fusion technology is applied, merging continuous operations such as convolution, batch normalization, and activation functions into a single kernel function, reducing memory access and data transfer overhead; automatic kernel tuning is performed for the Jetson Nano's Maxwell GPU architecture, generating a highly optimized inference engine. After optimization, the lightweight OpenPose model is compressed to approximately 15MB, achieving an inference speed of over 30 FPS; the TSA-Net model achieves an inference speed of over 25 FPS, meeting the real-time processing requirements.

[0101] Step 4.2: Edge Computing System Integration and Configuration

[0102] The deployment environment is set up at an outdoor ski resort with the following configuration: The Jetson Nano 4GB development board is secured in a waterproof and dustproof enclosure and powered by a 12V / 5A DC power supply; a high-speed camera is connected via USB 3.0, supporting 1080p resolution and 30 FPS video acquisition; a small display screen is connected via HDMI for real-time display of analysis results; an optional Bluetooth audio module is available for voice feedback. An Ubuntu 18.04 LTS system image (JetPack 4.6 SDK) is burned onto the Jetson Nano, installing CUDA 10.2, cuDNN v8, TensorRT 7.2 acceleration libraries, and Python 3.6, OpenCV 4.1, and PyTorch 1.10 (for debugging). The TensorRT engine file, model configuration file, and startup script generated in step 4.1 are burned to the device, and the system is set to start automatically upon power-up, ensuring that the model is automatically loaded and the system enters working mode.

[0103] Step 4.3: Real-time Inference and Multimodal Feedback Mechanism

[0104] At an outdoor ski resort, once a skier enters the camera's field of view, the system automatically initiates a real-time processing pipeline. The specific process is as follows: First, the camera captures 1080p video frames at 30 FPS and downsamples them to 368×368 as input to the lightweight OpenPose model; the model outputs in real-time coordinates of 17 human body key points and angles of joints such as hip, knee, shoulder, and elbow; the system draws a human skeleton diagram based on the key point coordinates and overlays and displays real-time joint angle values. Subsequently, continuous video frames containing skeleton information (every 32 frames constitute a segment) are fed into the TSA-Net model, and the model outputs a current motion quality score (0-100 points). The feedback mechanism includes three forms: 1) Visual feedback: Real-time display of skeletal visualization, joint angle values, and movement scoring curves on the screen; 2) Voice feedback: When key movement nodes (such as turns or jumps) are detected or the score falls below a preset threshold, prompts are broadcast via Bluetooth headsets, such as "Knee angle too large, please lower your center of gravity" or "Current movement score 82 points, continue"; 3) Data storage and remote transmission: Posture data, timestamps, and scoring results for each frame are recorded in JSON format to a local MicroSD card, and simultaneously uploaded to a cloud server via Wi-Fi using the MQTT protocol, allowing instructors to remotely view and analyze historical data. System latency is controlled within 50ms to ensure immediate feedback and provide skiers with real-time and effective training guidance.

[0105] In particular, in some preferred embodiments of the present invention, a computer device is also provided, including a memory and a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube described in any of the above embodiments.

[0106] In some other preferred embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube described in any of the above embodiments.

[0107] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above embodiments of the outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube, which will not be repeated here.

[0108] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for intelligent motion analysis of outdoor skiing based on lightweight OpenPose and spatiotemporal management, characterized in that, Includes the following steps: Acquire multi-angle training video data and extract video frame images; manually annotate the coordinates of key points and joint angles of the human body in each frame image, and score the motion quality to construct a dataset; A modified OpenPose network was constructed. The Darknet53 network with CSP modules introduced at the head end was used as the backbone network to extract features. Then, the key point heatmap and limb affinity field features were iteratively optimized through a multi-stage cascaded double-branch network to output the coordinates of human key points. Finally, the joint angle calculation branch embedded at the end of the modified OpenPose network was used to output the angles of each joint. A TSA-Net network is constructed; the TSA module is used to aggregate spatiotemporal features, and the target information of continuous frames is associated through the single target tracking method. The temporal coordinates are aggregated to construct a three-dimensional spatiotemporal tube, and the self-attention feature calculation is restricted from global dense calculation to sparse calculation within the tube. After being spliced ​​and fused with the key point coordinates and joint angles output by the modified OpenPose network, the motion quality score is output through a multilayer perceptron. The training images to be detected are sequentially input into the trained modified OpenPose network and TSA-Net network, and the action quality scores are output.

2. The outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube as described in claim 1, characterized in that, The key points include the head, chest, waist, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left foot, and right foot; the joint angles include the hip joint angle, knee joint angle, shoulder joint angle, and elbow joint angle.

3. The outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube as described in claim 1, characterized in that, The image is input into the backbone network CSP-Darknet53. After features are extracted through convolutional layers in the base layer, they are input into the CSP module. The CSP module splits the image into two branches along the channel dimension for separate processing. One branch goes directly to the output, while the other is fed into a dense block for deep feature extraction. The shallow detail features and deep semantic features are fused and then output after convolutional layers in subsequent layers to restore the number of channels. The image is then processed by the Darknet53 network to output multi-scale hierarchical features.

4. The outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube as described in claim 1, characterized in that, The multi-stage cascaded dual-branch network obtains candidate key point coordinates through heatmap peak retrieval and coordinate mapping processing, and completes human joint bipartite map matching and grouping by combining limb affinity field features, outputting two-dimensional key point coordinates.

5. The outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube according to claim 1, characterized in that, During the training phase, the parameters of the backbone network are first fixed, and only the multi-stage dual-branch iterative CNN module and the joint angle branch are trained; then the modified OpenPose network is jointly fine-tuned. The loss function for the training phase is: in, Loss for the two-dimensional probabilistic heatmap of key points; This results in the loss of some affinity domains. This is due to joint angle loss; , , These are the weighting coefficients.

6. The outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube according to claim 1, characterized in that, The TSA module performs spatiotemporal feature aggregation as follows: Step 1.1: Generate the bounding box of the skier in each frame of the video sequence using the target tracking algorithm, arrange the bounding boxes of all frames in chronological order, and perform Kalman filtering to smooth the sequence; stack the smoothed T bounding boxes on the time axis to form a three-dimensional spatiotemporal tube. Step 1.2: Using the I3D network as the backbone network, extract image features frame by frame through 3D convolutional kernels; map the spatiotemporal tube region onto the feature map scale, downsample the outer bounding box to obtain the spatial region on the corresponding feature map; perform self-attention feature interaction on the features within the spatial region; The formula for calculating the coordinate downsampling of the bounding rectangle onto the feature map is as follows: in, For the first The minimum column index on the feature map that the frame's bounding rectangle is mapped to. For the first The maximum column index on the feature map that the frame's bounding rectangle maps to; For the first The minimum row index on the feature map that the frame's bounding rectangle is mapped to; For the first The maximum row index on the feature map that the frame's bounding rectangle is mapped to; and For the first The horizontal and vertical coordinates of the center of the frame bounding rectangle in the original input frame; and For the first The width and height of the frame bounding rectangle in the original input frame; This represents the sampling multiple.

7. The outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube according to claim 1, characterized in that, The preprocessed keypoint coordinates and joint angles are concatenated, and then concatenated with the spatiotemporal features obtained by global average pooling to obtain the fused features; The fused features are input into a multilayer perceptron, which outputs a score for each action.

8. The outdoor skiing intelligent motion analysis method based on lightweight OpenPose and spatiotemporal tube according to claim 1, characterized in that, The loss function of the multilayer perceptron is: in, , These are the predicted and actual score distributions, respectively. The number of discretizations for the score levels; Indicates the first Each score level.

9. An outdoor skiing intelligent motion analysis system based on lightweight OpenPose and spatiotemporal management, characterized in that, Perform the method as described in any one of claims 1 to 8; The outdoor skiing intelligent motion analysis system includes: Data acquisition and preprocessing module: Completes multi-angle training video data acquisition, frame image extraction and annotation of key points and joint angles of the human body, as well as motion quality scoring; constructs dataset and performs unified preprocessing operations on images and labeled data within the dataset; Lightweight human keypoint detection module: The Darknet53 network with CSP module introduced at the head end is used as the backbone network to extract features. Then, the keypoint heatmap and limb affinity field features are iteratively optimized through a multi-stage cascaded double-branch network to output the coordinates of human keypoints. Then, the joint angle calculation branch embedded at the end of the modified OpenPose network is used to output the angles of each joint. The motion quality assessment module of the spatiotemporal tube: The TSA module is used to aggregate spatiotemporal features. The target information of continuous frames is associated through the single target tracking method. The temporal coordinates are aggregated to construct a three-dimensional spatiotemporal tube. The self-attention feature calculation is restricted from global dense calculation to sparse calculation within the tube. After being spliced ​​and fused with the key point coordinates and joint angles output by the modified OpenPose network, the motion quality score is output through the multilayer perceptron. Real-time feedback module: Provides real-time visualization results of the human skeleton and motion quality scores.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images