A method and device for analyzing and training athletes' throwing movements based on deep learning

Abstract

The present invention relates to the field of image processing, and discloses a method for analyzing and training an athlete's throwing action based on deep learning, comprising: using a sensor to synchronously collect standard throwing action data to construct a source domain data set; using a monocular camera to collect a throwing video to be analyzed in a target scene as target domain data; registering and fusing the source domain data to construct a complete and precise three-dimensional skeleton model of the athlete and a three-dimensional trajectory model of the thrown object; extracting two-dimensional key points of the athlete and two-dimensional positions of the thrown object from the target domain video; in a virtual environment similar to the target scene, importing the source domain standard throwing action three-dimensional model, and rendering to obtain throwing action images under different perspectives; designing an end-to-end throwing action analysis deep neural network model, using the trained model to infer the throwing video to be analyzed in the target domain, and generating text and sample video feedback.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for analyzing and training athletes' throwing movements based on deep learning. Background Art

[0002] Traditional methods for analyzing athletes' throwing movements, such as the patent application with publication number CN116665296A, disclose a method for identifying athletes' throwing movements based on the PCA-LBP algorithm. However, the method only identifies athletes' throwing movements and cannot associate the throwing movements with the motion trajectory of the thrown objects, thus lacking in guiding significance.

[0003] Therefore, it is necessary to propose a new type of athlete throwing action analysis and training method and equipment based on deep learning to solve the problem that the traditional athlete throwing action analysis method only recognizes the athlete's throwing action, resulting in the inability to associate the throwing action with the motion trajectory of the thrown object and lacks guidance. Summary of the invention

[0004] The present invention provides an athlete throwing action analysis and training method and equipment based on deep learning, which solves the problem that the traditional athlete throwing action analysis method only recognizes the athlete's throwing action, resulting in the inability to associate the throwing action with the motion trajectory of the thrown object and lacks guiding significance.

[0005] The technical problem solved by the present invention is achieved by adopting the following technical solutions:

[0006] A method for analyzing and training an athlete's throwing motion based on deep learning, characterized in that the method comprises:

[0007] Use sensors to synchronously collect standard throwing action data to construct a source domain dataset; use a monocular camera to collect the throwing video to be analyzed in the target scene as the target domain data;

[0008] The source domain data is registered and fused, and a complete and detailed 3D skeleton model of the athlete and the 3D trajectory model of the thrown object are constructed through a 3D reconstruction algorithm; the 2D key points of the athlete and the 2D position of the thrown object are extracted from the target domain video using a computer vision algorithm;

[0009] In a virtual environment similar to the target scene, the source domain standard throwing action 3D model is imported, and the throwing action images under different perspectives are obtained through virtual camera rendering;

[0010] Based on the real samples and virtual samples in the source domain and the key frames of the target domain video as training data, an end-to-end deep neural network model for throwing action analysis is designed; the deep neural network model adopts a multi-task joint training strategy, and the loss function includes the skeleton heat map regression loss, the throwing object bounding box regression loss, the throwing object classification loss and the key action segmentation loss, and the total loss is calculated according to the following formula:

[0011] ,in, For the key point detection loss, the MSE of the heat map and Gaussian GT map is used; For the geometric constraints of key points, the length of skeleton segments and the L1 loss of GT are used; For the throwing object box regression loss, smooth L1 loss is used; For the throwing object classification loss, the focal loss function is adopted; For the action segmentation loss, cross entropy is used. is the weight factor of each task loss;

[0012] Use the trained model to infer the throwing video to be analyzed in the target domain, give quantitative evaluation indicators, select the most similar standard throwing video, and generate text and example video feedback.

[0013] Furthermore, the use of sensors to synchronously collect standard throwing motion data includes: a binocular camera to obtain multi-view RGB video, a depth camera to obtain a human body depth map, and an inertial sensor to obtain three-dimensional posture information of key parts of the human body.

[0014] Furthermore, the collected source domain data is preprocessed, including: extracting SIFT features from RGB video frames, and using the RANSAC algorithm to complete the synchronization and registration of sequences of different viewpoints; performing distortion correction and spatial downsampling on the depth map, aligning it with the RGB map, and obtaining the registered RGB-D sequence; performing time synchronization, coordinate system alignment and Kalman filtering on the inertial sensor data to obtain a smooth and continuous three-dimensional joint motion curve.

[0015] Furthermore, the target domain video data is preprocessed: the OpenPose tool is used to extract the two-dimensional skeleton key points of the athlete in each frame of the image; the Faster R-CNN target detection algorithm is used to locate the pixel coordinates of the thrown object in the image and mark the timestamp of the frame in which it is located.

[0016] Furthermore, the construction of the three-dimensional skeleton model of the athlete includes: sending the aligned RGB-D sequence into a three-dimensional reconstruction pipeline based on a voxel occupancy map, which first converts each frame of the RGB-D image into a probabilistic voxel grid, and then reconstructs a continuous and complete three-dimensional human surface model through a credibility-weighted TSDF fusion method, and the fused model is represented as a high-resolution voxel grid; using a human posture estimation algorithm on the RGB image to obtain a three-dimensional human skeleton of each frame, and the skeleton structure is defined as a tree topology of a number of joints and a number of connecting rods; aligning the estimated skeleton with the reconstructed surface model, and using the joints in the skeleton as deformation controls to drive the surface mesh to move.

[0017] Furthermore, a three-dimensional trajectory model of the projectile is constructed, including: detecting and segmenting the projectile area from the RGB video to form a binary mask; using the Lucas-Kanade optical flow algorithm on consecutive frames to track the pixel coordinates of the projectile; using the projectile mask to intercept the corresponding area from the RGB image and the depth image to obtain the RGB-D subimage of the projectile, and converting the subimage into a point cloud representation; assuming that the projectile is approximately a regular geometric shape, using the RANSAC algorithm to fit the shape parameters of the point cloud; averaging the parameter results estimated in different frames to obtain the three-dimensional size of the projectile; using the Kalman filter algorithm on consecutive frames to track the three-dimensional center of mass coordinates of the projectile to obtain its complete flight trajectory.

[0018] Furthermore, the throwing action images under different perspectives are obtained by virtual camera rendering, including: arranging a plurality of virtual cameras in a virtual environment to evenly cover all directions of the throwing action, mapping the joint angles of the athlete's three-dimensional skeleton model to the skeleton controller of the virtual character, and automatically generating a plurality of variant actions based on motion capture data.

[0019] Furthermore, the neural network model takes the video clip as input and outputs the corresponding three-dimensional throwing trajectory, human skeleton movement parameters and key action time points.

[0020] A deep learning-based athlete throwing motion analysis and training device comprises: a motion data acquisition device, a data processing terminal and a model training server, so as to realize the above-mentioned deep learning-based athlete throwing motion analysis and training method.

[0021] The present invention has the following beneficial effects:

[0022] The invention uses a sensor to synchronously collect standard throwing action data to construct a source domain data set; uses a monocular camera to collect a throwing video to be analyzed in a target scene as target domain data; performs registration and fusion on the source domain data, and constructs a complete and detailed three-dimensional skeleton model of an athlete and a three-dimensional trajectory model of a thrown object through a three-dimensional reconstruction algorithm; uses a computer vision algorithm to extract two-dimensional key points of the athlete and two-dimensional positions of the thrown object from the target domain video; imports the source domain standard throwing action three-dimensional model in a virtual environment similar to the target scene, and obtains throwing action images under different viewing angles through virtual camera rendering; designs an end-to-end throwing action analysis deep neural network model based on the source domain real samples and virtual samples, and the target domain video key frames as training data, uses the trained model to infer the target domain throwing video to be analyzed, gives quantitative evaluation indicators, selects the most similar standard throwing video, generates text and sample video feedback, assists athletes in optimizing action essentials, has a strong training guidance function, and solves the problem that the traditional athlete throwing action analysis method is only for the recognition of the athlete's throwing action, resulting in the inability to associate the throwing action with the motion trajectory of the thrown object and lacks guidance significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0025] Figure 2 A wireframe diagram of the device of the present invention. DETAILED DESCRIPTION

[0026] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0027] Please refer to Figure 1 , Figure 1 A flowchart of an athlete throwing motion analysis and training method based on deep learning is provided in an embodiment of the present invention.

[0028] like Figure 1 As shown, the athlete throwing action analysis and training method 100 based on deep learning includes steps 101 to 105.

[0029] Step 101, multi-source heterogeneous data collection. Use multiple types of sensors to synchronously collect standard throwing action data, including binocular cameras to obtain multi-view RGB videos, depth cameras to obtain human depth maps, inertial sensors to obtain 3D posture information of key parts of the human body, etc., to build a source domain dataset. Use a monocular camera to collect the throwing video to be analyzed in the target scene as the target domain data.

[0030] Exemplarily, five professional athletes are selected, each of whom repeats a standard throwing action 50 times, and the motion data is synchronously collected using the sensor device of the motion data collection device.

[0031] Specifically, two high-speed cameras are used to face the front and back of the athlete respectively, to collect 1080p, 60fps RGB video sequences. The relative pose between the two cameras is known and can be obtained through Zhang's camera calibration method. Inertial sensors are strapped on six key parts of the athlete: shoulder, elbow, wrist, hip, knee, and ankle. The sensor is synchronized with the data processing terminal through a Bluetooth module to collect the three-dimensional angular velocity and acceleration of each joint at a frequency of 100Hz. A TOF depth camera is used to collect the scene depth map at a frequency of 30fps and align it with the extrinsic parameters of the RGB camera. The depth map is aligned with the RGB Figure 1 A total of 250 throwing samples were obtained, each of which contained 2 1080p RGB videos, 1 1024*1024 depth video, and inertial data of 6 joint positions, which were used for the subsequent source domain 3D model construction.

[0032] Furthermore, using a common monocular camera, 30 amateur athletes' throwing videos are randomly collected in the target scene, 1-2 samples per person, and a total of 50 samples are obtained. The acquisition equipment parameters are 720p, 30fps. The collected videos will be used to extract the two-dimensional key points of the athletes and the positions of the thrown objects as the target domain training data.

[0033] Exemplarily, the collected source domain data is preprocessed: the scale-invariant feature transform (SIFT) features are extracted from the RGB video frames, and the random sampling consistency (RANSAC) algorithm is used to complete the synchronization and alignment of different perspective sequences; the depth map is distortion corrected and spatially downsampled, and aligned with the RGB map to obtain the aligned RGB-D sequence; the inertial sensor data is time synchronized, the coordinate system is aligned and Kalman filtered to obtain a smooth and continuous three-dimensional joint motion curve.

[0034] Exemplarily, the target domain video data is preprocessed as follows: the two-dimensional skeletal key points (shoulder, elbow, wrist, hip, knee, ankle) of the athlete in each frame are extracted using tools such as OpenPose, and missed or misdetected frames are manually corrected; target detection algorithms such as Faster R-CNN are used to locate the pixel coordinates of the thrown object in the image, and the timestamp of the frame in which it is located is marked.

[0035] After the above preprocessing steps, the source domain data forms a tightly aligned "RGB video-depth video-inertial data" triplet, which can be used for high-precision three-dimensional motion reconstruction; the target domain data forms a timestamped "skeletal key point-throwing object coordinate" sequence, which can be used to assist the training of deep learning models.

[0036] Step 102, 3D motion reconstruction. The source domain data is registered and fused, and a complete and detailed 3D skeleton model of the athlete and the 3D trajectory model of the thrown object are constructed through a 3D reconstruction algorithm. The 2D key points of the athlete and the 2D position of the thrown object are extracted from the target domain video using a computer vision algorithm.

[0037] Exemplarily, the source domain multimodal data collected and preprocessed by the motion data acquisition device is used to construct a fine geometric model and a skeletal motion model of the athlete through a three-dimensional reconstruction algorithm.

[0038] Specifically, the registered RGB-D sequence is fed into a 3D reconstruction pipeline based on voxel occupancy maps. The pipeline first converts each frame of RGB-D image into a probabilistic voxel grid, and then reconstructs a continuous and complete 3D human surface model through a credibility-weighted truncated signed distance function (TSDF) fusion method. The fused model is represented as a high-resolution voxel grid. The human pose estimation algorithm is used on the RGB image to obtain the 3D human skeleton of each frame. The skeleton structure is defined as a tree topology of 15 joints (head, neck, shoulder × 2, elbow × 2, wrist × 2, hip, knee × 2, ankle × 2) and 14 links (head-neck, neck-trunk, trunk-hip, thigh × 2, calf × 2, upper arm × 2, lower arm × 2). The estimated skeleton is aligned with the reconstructed surface model, and the joints in the skeleton are used as deformation controls to drive the surface mesh to move. Specifically, the LBS (Linear Blend Skinning) algorithm is used to calculate the weight of each surface vertex to the skeleton joint. When the joint moves, the vertex follows the rigid body transformation to achieve mesh deformation. At the same time, the data of 6 inertial sensors are introduced to optimize the skeleton movement. Specifically, the RGB video and inertial data are aligned on the time axis, the three-dimensional position of the sensor is mapped to the skeleton joint, the acceleration and angular velocity measured by inertia are used to construct the motion prior, and the interactive probability clearing algorithm (ICP) algorithm is used to iteratively optimize the motion trajectory of the skeleton to minimize the inverse kinematics error.

[0039] Furthermore, standard projectiles (such as shot put and discus) are reconstructed in three dimensions to obtain their detailed geometric shape and motion trajectory. The projectile area is detected and segmented from the RGB video to form a binary mask. The Lucas-Kanade optical flow algorithm is used to track the pixel coordinates of the projectile on consecutive frames. Using the projectile mask, the corresponding area is intercepted from the RGB image and the depth image to obtain the RGB-D sub-image of the projectile, and the sub-image is converted into a point cloud representation. Assuming that the projectile is approximately a regular geometric shape (sphere, disk), the RANSAC algorithm is used to fit the shape parameters of the point cloud (such as the center of the sphere and the radius). The parameter results estimated in different frames are averaged to obtain the three-dimensional size of the projectile. The Kalman filter algorithm is used to track the three-dimensional center of mass coordinates of the projectile on consecutive frames to obtain its complete flight trajectory.

[0040] After the above steps, a high-fidelity geometric model of the athlete and the thrown object in three-dimensional space and a continuous motion sequence can be obtained. Compared with traditional two-dimensional image analysis, three-dimensional reconstruction can provide accurate scale and angle information of the object, laying a solid data foundation for subsequent motion analysis.

[0041] Step 103, virtual sample generation: In a virtual environment similar to the target scene, a standard throwing action 3D model of the source domain is imported, and a large number of throwing action images under different perspectives are obtained through virtual camera rendering to enrich the quantity and diversity of source domain training samples.

[0042] Exemplarily, to expand the training data of the deep learning model and improve the generalization ability of the algorithm in different scenarios, a large number of throwing action samples need to be generated in a virtual environment similar to the target domain. According to the environmental requirements of the throwing sport, the ground material (such as concrete, plastic track, lawn, etc.), wind force, lighting and other physical conditions of the virtual scene are designed. Different materials correspond to different texture maps and reflective properties. Wind force and lighting can be controlled by parameters to simulate the impact of throwing actions in different weather. Design a variety of competition venue layouts, including standard throwing areas, spectator seats and other elements, and add certain background debris (such as billboards, flags, etc.). By changing the position, quantity, type, etc. of these elements, a variety of scenes can be constructed. Arrange a large number of virtual cameras in the virtual scene to simulate different shooting angles and distances. The internal and external parameters of the camera can be randomly generated, covering various common shooting devices (mobile phones, cameras) and shooting postures (looking straight, looking up, looking down).

[0043] Furthermore, based on the above principles, a flexible and configurable virtual environment for throwing sports is constructed using a three-dimensional rendering engine (such as Unity3D). The environment supports real-time rendering and dynamic adjustment of parameters, and can easily generate various derivative scenes.

[0044] Furthermore, the above-constructed three-dimensional models of athletes and projectiles are imported into a virtual environment. Through the action redirection technology, the model is made to perform throwing actions at various predefined speeds and forces, and multi-view RGB images are rendered from different perspectives. 100 virtual cameras are arranged in the virtual environment to evenly cover all directions of the throwing action. The internal parameters of each camera are set to common consumer-grade camera parameters (resolution 1920x1080, field of view 60°). The joint angles of the athlete's three-dimensional skeleton model are mapped to the skeleton controller of the virtual character. Based on the motion capture data, 10 variant actions (such as knee bending degree, throwing height, holding time, etc.) are automatically generated. The variation amplitude is controlled within ±10% of the original action. Add a projectile model to the hand joints of the athlete's three-dimensional model. Add appropriate highlights and roughness to the material properties of the projectile model to simulate real reflections and textures. Set the physical parameters of the projectile (mass, initial velocity, drag coefficient, etc.) and simulate its parabolic motion trajectory in the virtual environment. According to Newton's laws of mechanics, calculate the three-dimensional coordinates, rotation angle and other state quantities of the projectile at each moment. Set the rendering parameters of the virtual camera, including frame rate (60fps), exposure time (1 / 250s), depth of field (f / 8), etc. In each variant action cycle, all cameras are triggered to shoot synchronously at a fixed time interval (such as 5 frames) to obtain a total of 12,000 renderings. Add data enhancement to the renderings in batches, including Gaussian blur, motion blur, noise, etc., to simulate the image quality degradation under real shooting conditions. The enhanced renderings are mixed with the original images in a 1:1 ratio.

[0045] After the above process, more than 10,000 multi-view throwing action images can be derived from a single motion capture data, covering different action variants and shooting conditions, which can effectively alleviate the demand for large-scale real samples for model training.

[0046] Step 104, end-to-end model training. Based on the source domain real samples and virtual samples, and the target domain video key frames as training data, an end-to-end throwing action analysis deep neural network model is designed. The model takes the video clip as input and outputs the corresponding three-dimensional throwing trajectory, human skeleton motion parameters and key action time points.

[0047] Exemplarily, the source domain real throwing video collected by the motion data acquisition device, the generated source domain virtual throwing video, and the target domain throwing video are respectively constructed as a training set and a test set. The specific scheme is as follows:

[0048] Specifically, the throwing videos of 10 athletes were selected as the training set, with 20 actions per person and 100 frames per action, for a total of 20,000 frames. 50,000 frames were randomly selected from the virtual rendering data to supplement the training set, and the source domain training set totaled 70,000 frames. The videos of another 2 athletes were used as the source domain test set, for a total of 4,000 frames. The throwing videos of 15 athletes were selected as the training set, for a total of 15,000 frames. The videos of the remaining 5 athletes were used as the target domain test set, for a total of 5,000 frames. For each frame of the image, the coordinates of the athlete's 15 skeletal key points (head, neck, shoulder, elbow, wrist, hip, knee, ankle), the coordinates of the 5 key points of the thrown object (center of the ball, leftmost, rightmost, topmost, bottommost) and the visibility mark were annotated.

[0049] In addition, the video sequence is divided into six stages according to the rules of throwing actions (preparation, run-up, charging, acceleration, release, and closing), and the start and end time points of each stage are marked.

[0050] Furthermore, the present invention designs an end-to-end deep learning model for throwing action analysis, named TDNet. The model takes a continuous RGB image sequence as input, and outputs a sequence of skeletal key points of the throwing athlete, a sequence of spatial coordinates of the thrown object, and key action time points. Using 3D CNN as the backbone network, convolution is performed simultaneously in the time dimension and the spatial dimension to extract the spatiotemporal features of the image sequence. The backbone network contains 4 consecutive 3D convolutional layers and 3D pooling layers, and finally outputs a feature map of size 256x8x8. Three transposed convolutional layers are connected in parallel on the feature map to restore the bone heat map within the three scales of the whole body, upper limbs, and lower limbs respectively. The heat map resolutions are 64x64, 128x128, and 256x256, respectively. The heat map results of the three scales are weighted averaged to obtain the final skeletal key point prediction. The feature map is input into a region proposal network (RPN) to extract 128 candidate boxes of the thrown object. The candidate box is subjected to bounding box regression and classification, and the spatial coordinates and confidence of the thrown object are output. The top 5 confidence results are used as the final output. The feature map is passed through the Bi-LSTM layer to automatically learn the timing dependencies of key actions. The LSTM output is connected to the 1D convolution layer and the sigmoid function to output a 6-channel action phase probability sequence. The probability sequence is thresholded to obtain the time boundary of the key action.

[0051] Furthermore, TDNet adopts a multi-task joint training strategy. The loss function includes the skeleton heat map regression loss, the thrown object bounding box regression loss, the thrown object classification loss and the key action segmentation loss. The total loss is calculated according to the following formula:

[0052]

[0053] in, For the key point detection loss, the mean squared error (MSE) of graphical trawl (GT) using heatmap and Gaussian graph is used; For the geometric constraints of key points, the length of the skeleton segment and the L1 loss of GT are used; For the throwing object box regression loss, smooth L1 loss is used; For the object classification loss, the focal loss function is used; For the action segmentation loss, cross entropy is used. is the weight factor of each task loss. The model optimization uses the Adam optimizer with segmented learning rate adjustment, and the initial learning rate is set to 1e-4. The model performance is evaluated on the validation set after each stage. If the performance does not improve for 5 consecutive stages, the learning rate is divided by 2, and the minimum learning rate is not less than 1e-6. A total of 100 stages are trained.

[0054] Furthermore, the model training is divided into two stages: pre-training stage and joint optimization stage. The skeleton key point detection and thrown object detection modules of TDNet are trained using real samples from the source domain and virtual samples respectively, and the parameters of the two modules are not shared. 50 stages are trained on real samples and 100 stages are trained on virtual samples. The pre-training stage is used to extract a general throwing motion representation. Based on the pre-training parameters, the entire TDNet is trained end-to-end using mixed data from the source domain and target domain training sets. The mixing ratio of real samples and virtual samples increases linearly from 1:5 to 2:1 to alleviate the distribution difference between domains. The target domain data only participates in the training of skeleton key point detection and thrown object detection, and does not participate in action segmentation training. The joint optimization stage improves the generalization performance of the model by jointly learning the multi-dimensional representation of throwing motion.

[0055] Step 105: Action analysis and feedback. Use the trained model to infer the target domain throwing video to be analyzed, give quantitative evaluation indicators, select the most similar standard throwing video, generate text + sample video feedback, and assist athletes in optimizing action essentials.

[0056] For example, the trained TDNet model is used to perform forward reasoning on the newly collected throwing videos in the target domain to obtain the prediction results of the athlete's skeleton key points, the spatial position of the thrown object, and the key action time points in the video. On this basis, a series of quantitative indicators are designed to evaluate the completion quality of the throwing action from multiple angles:

[0057] (1) Throwing angle: Calculate the spatial vector angle between the shoulder, elbow, and wrist when the projectile leaves the hand, compare it with the throwing angle of excellent athletes, and quantify the degree of standardization of the action.

[0058] (2) Throwing speed: Based on the spatial displacement of the thrown object in adjacent frames and the frame rate, the instantaneous speed and average speed of the thrown object are estimated and compared with the standard action.

[0059] (3) Throwing height: Take the Z coordinate of the highest point of the projectile's trajectory, subtract the height of the throwing starting point, and obtain the vertical throwing height, which is normalized with the athlete's height.

[0060] (4) Number of steps during the run-up: Based on the displacement changes of key points of the lower limbs, the stepping action during the run-up is detected, the total number of steps is calculated, and compared with the run-up rhythm of excellent athletes.

[0061] (5) Charging time: Based on the detection results of key action time points, calculate the duration of the charging action before throwing to evaluate whether the best charging effect is achieved.

[0062] (6) Overall fluency: The DTW distance between the key point trajectory and the standard action is used as a quantitative indicator of overall fluency. The smaller the distance, the higher the fluency.

[0063] Furthermore, in the action library of excellent athletes in the source domain, several action clips that are most similar to the action to be analyzed in the target domain are retrieved as demonstration samples of feedback prompts. Based on the skeleton key point sequence extracted by TDNet, a spatiotemporal feature representation is constructed for all action clips in the source domain action library, and each clip is represented by a feature vector of fixed dimension. The same feature representation is extracted for the target domain clip to be analyzed, and the feature space Euclidean distance between it and the source domain action clip is calculated, and the Top-K clips with the closest distance are selected as similar actions. The skeleton sequences of the source and target domains are uniformly normalized in time and space, and the dynamic time warping (DTW) algorithm is used to calculate the optimal alignment of the two sequences and quantify the overall similarity of the sequences. The two indicators of feature space distance and DTW distance are combined to reorder the Top-K similar clips to form a progressive feedback sequence from similar to different from the action to be analyzed.

[0064] Furthermore, targeted text feedback is given for the quantitative indicators extracted from the action analysis indicators, including the differences from the standard actions, the possible changes in throwing results, etc.; at the same time, similar actions retrieved in the similar action retrieval are presented to intuitively show users the essentials of standard actions.

[0065] Furthermore, the quantitative indicators of the action to be analyzed are compared with the standard action indicators in the source domain one by one, and the absolute and relative differences are calculated. The difference amplitude is graded (such as ±5% is excellent, ±5%~±15% is medium, and more than ±15% is unqualified), and an evaluation statement for each indicator is generated. Based on the logical relationship between the indicator value and the threshold, the templates matching each evaluation statement are selected from the predefined feedback sentence templates, and the values ​​therein are replaced to form a complete feedback statement. The feedback statements of each indicator are sorted in order of importance such as overall fluency, throwing speed, throwing angle, etc., and organized into a structured feedback report.

[0066] Furthermore, the original RGB video frames of the retrieved Top-K similar clips are decoded and input into TDNet together with the video to be analyzed to extract their respective skeleton key point sequences. The source domain similar video and the target domain video are aligned in time and space scales, and the relative position of the skeleton in each frame remains consistent. The source domain similar video and the target domain video are placed side by side and played synchronously. The skeleton is drawn on the original RGB frame using alpha blending to intuitively present the similarities and differences in the action execution process. Text feedback matching the current action stage is scrolled below the video frame to prompt the user where to focus.

[0067] After the above process, the quantitative evaluation index of the throwing action to be analyzed and the examples of similar standard actions can be presented to the user. The user can understand the advantages and disadvantages of his own action through text descriptions, and intuitively feel the direction of improvement through video comparison, thus forming objective, comprehensive and easy-to-understand action feedback.

[0068] Please refer to Figure 2 , Figure 2 A method for analyzing and training an athlete's throwing motion based on deep learning is proposed. The hardware equipment of the athlete's throwing motion analysis and training device 200 based on deep learning mainly includes: a motion data acquisition device 201, a data processing terminal 202 and a model training server 203.

[0069] Furthermore, the motion data acquisition device 201 includes: 2 industrial-grade high-speed cameras for multi-view RGB video acquisition, the reference model can be Basler acA1920-155uc, with a resolution of 1920*1080 and a frame rate of 60fps; 6 inertial sensors for collecting three-dimensional posture information of key parts of the human body. Each sensor has a built-in three-axis accelerometer, gyroscope and magnetometer. 1 TOF depth camera for collecting human body depth map, the reference model can be Kinect Azure, with a resolution of 1024*1024 and a frame rate of 30fps; 1 ordinary monocular camera for target domain throwing video acquisition.

[0070] The data processing terminal 202 is used to align, fuse and preprocess the collected multi-source data, and upload the processed data to the model training server. The terminal is configured as a mainstream workstation, which can include an Intel i7 processor, 32GB memory, NVIDIA RTX 2080Ti graphics card, etc.

[0071] The model training server 203 is used to perform computationally intensive tasks such as virtual sample generation, deep learning model training and reasoning. The server is configured as a high-performance GPU workstation, which may include a 2-way Intel Xeon Gold 6230R processor, 256GB memory, 8 NVIDIA Tesla V100 graphics cards, etc.

[0072] Figure 2 The hardware composition and system architecture of the present invention are demonstrated. The hardware devices are divided into three levels according to their uses: sensor devices, processing terminals, and servers. Among them, the sensor devices are responsible for collecting throwing motion data from different dimensions, the processing terminals are responsible for data fusion and preprocessing, and the servers are responsible for computationally intensive modeling, training, and analysis tasks. This hierarchical and heterogeneous system architecture can give full play to the strengths of various types of hardware, and can not only efficiently complete data collection and fusion, but also meet the computing power requirements of model training.

[0073] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for analyzing and training athletes' throwing movements based on deep learning, characterized in that: The methods include: Use sensors to synchronously collect standard throwing action data to construct a source domain dataset; use a monocular camera to collect the throwing video to be analyzed in the target scene as the target domain data; The source domain data is registered and fused, and a complete and detailed 3D skeleton model of the athlete and a 3D trajectory model of the thrown object are constructed through a 3D reconstruction algorithm; the construction of the 3D skeleton model of the athlete includes: sending the registered RGB-D sequence into a 3D reconstruction pipeline based on a voxel occupancy map, the pipeline first converts each frame of the RGB-D image in the RGB-D sequence into a probabilistic voxel grid, and then reconstructs a continuous and complete 3D human surface model through a credibility-weighted TSDF fusion method, and the fused model is represented as a high-resolution voxel grid; the RGB image intercepted from the RGB video is used with a human posture estimation algorithm to obtain a 3D human skeleton of each frame, and the skeleton structure is defined as a tree topology of a plurality of joints and a plurality of connecting rods; the estimated 3D human skeleton is aligned with the reconstructed surface model, and the joints in the skeleton are used as deformation controls to drive the 3D human skeleton. The surface mesh generates motion; the construction of the three-dimensional trajectory model of the projectile includes: detecting and segmenting the projectile area from the RGB video to form a binary mask of the projectile; using the Lucas-Kanade optical flow algorithm on continuous frames to track the pixel coordinates of the projectile; using the binary mask of the projectile, intercepting the corresponding area from the RGB image and depth image intercepted from the RGB video, obtaining the RGB-D sub-image of the projectile, and converting the sub-image into a point cloud representation; assuming that the projectile is approximately a regular geometric shape, using the RANSAC algorithm to fit the shape parameters of the point cloud; averaging the parameter results estimated in different frames to obtain the three-dimensional size of the projectile; using the Kalman filtering algorithm on continuous frames to track the three-dimensional center of mass coordinates of the projectile to obtain its complete flight trajectory; using a computer vision algorithm to extract the two-dimensional key points of the athlete and the two-dimensional position of the projectile from the target domain video; In a virtual environment similar to the target scene, the source domain standard throwing action 3D model is imported, and the throwing action images under different perspectives are obtained through virtual camera rendering; Based on the real samples and virtual samples in the source domain and the key frames of the target domain video as training data, an end-to-end deep neural network model for throwing action analysis is designed; the deep neural network model adopts a multi-task joint training strategy, and the loss function includes the skeleton heat map regression loss, the throwing object bounding box regression loss, the throwing object classification loss and the key action segmentation loss. The total loss is calculated according to the following formula: L total =λ1*(L kp-dt +L kp-hmgt )+λ2*(L obj-reg +L obj-cls )+λ3*L act , where L kp-dt is the key point detection loss, using the MSE of the heat map and Gaussian GT map; L kp-hmgt For the geometric constraints of key points, the length of the skeleton line segment and the L1 loss of GT are used; L obj-reg is the throwing object box regression loss, using smooth L1 loss; L obj-cls is the throwing object classification loss, using the focal loss function; L act is the action segmentation loss, using cross entropy, and λ is the weight factor of each task loss; Use the trained model to infer the throwing video to be analyzed in the target domain, give quantitative evaluation indicators, select the most similar standard throwing video, and generate text and example video feedback.

2. The method for analyzing and training athletes' throwing movements based on deep learning according to claim 1, characterized in that: The method of synchronously collecting standard throwing motion data using sensors includes: using a binocular camera to obtain multi-view RGB video, using a depth camera to obtain a human body depth map, and using an inertial sensor to obtain three-dimensional posture information of key parts of the human body.

3. The method for analyzing and training athletes' throwing movements based on deep learning according to claim 2, characterized in that: The preprocessing of the collected source domain data set includes: extracting SIFT features from RGB video frames, using the RANSAC algorithm to complete the synchronization and alignment of sequences of different viewpoints; performing distortion correction and spatial downsampling on the depth map, aligning it with the RGB image captured from the RGB video, and obtaining the aligned RGB-D sequence; performing time synchronization, coordinate system alignment and Kalman filtering on the three-dimensional posture information to obtain a smooth and continuous three-dimensional motion curve of the joint.

4. The method for analyzing and training athletes' throwing movements based on deep learning according to claim 1, characterized in that: Preprocess the target domain data: Use the OpenPose tool to extract the two-dimensional skeleton key points of the athlete in each frame of the image; use the Faster R-CNN target detection algorithm to locate the pixel coordinates of the thrown object in the image and mark the timestamp of the frame in which it is located.

5. The method for analyzing and training athletes' throwing movements based on deep learning according to claim 1, characterized in that: The throwing action images under different viewing angles are obtained by virtual camera rendering, including: arranging a plurality of virtual cameras in a virtual environment to evenly cover all directions of the throwing action, mapping the joint angles of the athlete's three-dimensional skeleton model to the skeleton controller of the virtual character, and automatically generating a plurality of variant actions based on motion capture data.

6. The method for analyzing and training athletes' throwing movements based on deep learning according to claim 1, characterized in that: The neural network model takes video clips as input and outputs corresponding three-dimensional throwing trajectories, human skeleton movement parameters and key action time points.

7. A deep learning-based athlete throwing motion analysis and training device, characterized in that: The athlete throwing motion analysis and training device based on deep learning includes: a motion data acquisition device, a data processing terminal and a model training server, so as to implement the athlete throwing motion analysis and training method based on deep learning as described in any one of claims 1 to 6.

Images