Deep learning-based intelligent identification method and system for air juggling action of table tennis

Abstract

The invention provides an intelligent identification method and system for an air juggling action of a table tennis ball based on deep learning. The method comprises the following steps: step 1, collecting real-time video data and attitude sensor data of air juggling through a mobile terminal; 2, constructing a deep learning action recognition model; 3, optimizing the pre-trained YOLO model by adopting a transfer learning strategy; 4, according to the relative position relation between the racket and the table tennis ball, a multi-constraint-condition ball hitting judgment algorithm is designed; and 5, feeding back an identification result in real time through the mobile terminal, wherein the identification result comprises error action correction demonstration, training report generation and quantitative index display. According to the method, the computer vision technology and the recognition algorithm are utilized, real-time recognition and analysis of the air juggling actions of the teenagers at home are achieved through mobile clients such as mobile phone applets, professional training guidance and feedback are provided for the teenagers, the teenagers are helped to standardize the actions, and the training effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning and sphere motion recognition. Specifically, it relates to an intelligent recognition method and system for table tennis air-bouncing actions based on deep learning. Background Art

[0002] Table tennis is deeply loved by the vast number of teenagers. However, traditional table tennis training requires a specific venue and a partner, which greatly limits the practice of teenagers during home stay. Air-bouncing is an important way of basic table tennis training, but currently there is a lack of effective home supervision and guidance means. When teenagers practice air-bouncing alone, they cannot know whether their actions are standard. Long-term incorrect actions may lead to slow improvement of skills or even the formation of bad habits. The existing action recognition technology is not yet perfect in the application of sports training, especially in the scenario of table tennis air-bouncing. It is difficult to accurately recognize and analyze complex and changeable table tennis bouncing actions, and cannot meet the needs of teenagers' efficient home training.

[0003] Patent application document CN110496377A discloses a virtual table tennis player hitting training method based on reinforcement learning, including the steps of: 1) designing a task scenario and a task process; 2) training the hitting strategy of the racket using the method of reinforcement learning; 3) estimating the movement of each joint of the human body when hitting the ball using the algorithm of inverse kinematics; 4) training the movement strategy of the root node using reinforcement learning. However, this patent cannot completely solve the existing technical problems and also cannot meet the requirements of the present invention. Summary of the Invention

[0004] Aiming at the defects in the prior art, the purpose of the present invention is to provide an intelligent recognition method and system for table tennis air-bouncing actions based on deep learning.

[0005] According to the intelligent recognition method for table tennis air-bouncing actions based on deep learning provided by the present invention, it includes:

[0006] Step 1: Collect real-time video data and attitude sensor data of air-bouncing through a mobile terminal;

[0007] Step 2: Construct a deep learning action recognition model, which is composed of a CNN module, an RNN module and a YOLO target detection module. Among them: the CNN module is used to extract the spatial features in the video frame, including the position information of the racket, the table tennis ball and the human body key points; the RNN module is used to capture the temporal correlation of the actions in the continuous video frames; the YOLO target detection module is based on the YOLOv8 architecture, realizes multi-scale feature fusion through the C2f module, and aggregates global context information through the SPPF module;

[0008] Step 3: Optimize the pre-trained YOLO model using transfer learning strategies, including weight transfer, fixing the feature extractor, and fine-tuning the classifier;

[0009] Step 4: Design a hitting judgment algorithm with multiple constraints based on the relative position relationship between the racket and the table tennis ball, including: determining the hitting action through peak detection of the vertical distance between the ball and the racket; combining the effectiveness judgment of the ROI area of the racket in the picture and the hitting frequency analysis;

[0010] Step 5: Real-time feedback the recognition results through the mobile terminal, including demonstration of correcting wrong actions, generation of training reports, and display of quantitative indicators.

[0011] Preferably, the CNN module includes:

[0012] The backbone network, which is composed of CBS modules, C2f modules, and SPPF modules. Among them, the CBS module is used to perform convolution on the input image or feature map, the C2f module fuses shallow detail features and deep semantic features through cross-layer connections, and the SPPF module aggregates the global context information of the table tennis ball juggling action;

[0013] The Neck layer network, which fuses multi-scale feature maps using upsampling and Concat operations;

[0014] The Head layer network, which adopts a decoupled head structure to generate classification feature maps and bounding box prediction feature maps respectively.

[0015] Preferably, the transfer learning strategy includes:

[0016] Use the pre-trained YOLOv8 model as the source model and adjust the output dimension of its fully connected layer to match the classification categories of the table tennis ball juggling action;

[0017] Add an RNN layer to the source model to enhance the ability to extract temporal features;

[0018] [[ID=зо]]Adopt the cross-entropy loss function and the CIoU loss function to jointly optimize the model parameters. The formula is:

[0019] L = λ1·L cls + λ2·L CIou

[0020] Where L cls is the classification loss, L CIou is the bounding box regression loss, and λ1, λ2 are weight coefficients.

[0021] Preferably, the hitting judgment algorithm further includes:

[0022] Define the effective hitting area, and determine an invalid hit when the center of the ball exceeds the preset proportional range of the racket bounding box;

[0023] Identify missed balls or abnormal ball juggling actions through the acceleration change of the sphere's movement trajectory;

[0024] Based on the Numpy module, count the number of times the spike signal exceeds the threshold, and calculate the number of ball juggling times per unit time.

[0025] Preferably, the real-time feedback of the mobile terminal is specifically implemented as follows:

[0026] Package the HTTP interface through FastAPI, receive the video stream, and return the marked result video and the counting data in JSON format;

[0027] Use the imageio library to encode the inference result into an MP4 file in H.264 format, supporting real-time playback on the web page;

[0028] Dynamically display the comparison between the correct action and the wrong action in the video in the form of animated overlay.

[0029] According to the intelligent recognition system for table tennis air ball juggling actions based on deep learning provided by the present invention, it includes:

[0030] Module M1: Collect real-time video data and attitude sensor data of air ball juggling through a mobile terminal;

[0031] Module M2: Build a deep learning action recognition model, which consists of a CNN module, an RNN module, and a YOLO object detection module. Among them: the CNN module is used to extract the spatial features in the video frame, including the position information of the racket, the table tennis ball, and the human key points; the RNN module is used to capture the temporal correlation of actions in consecutive video frames; the YOLO object detection module is based on the YOLOv8 architecture, realizes multi-scale feature fusion through the C2f module, and aggregates global context information through the SPPF module;

[0032] Module M3: Optimize the pre-trained YOLO model using transfer learning strategies, including weight transfer, fixing the feature extractor, and fine-tuning the classifier;

[0033] Module M4: Design a batting judgment algorithm with multiple constraint conditions according to the relative position relationship between the racket and the table tennis ball, including: determining the batting action through the spike detection of the vertical distance between the ball and the racket; combining the effectiveness judgment of the ROI area of the racket in the picture and the batting frequency analysis;

[0034] Module M5: Real-time feedback the recognition result through the mobile terminal, including demonstration of correcting wrong actions, generation of training reports, and display of quantitative indicators.

[0035] Preferably, the CNN module includes:

[0036] The backbone network is composed of CBS modules, C2f modules, and SPPF modules. Among them, the CBS module is used to perform convolution on the input image or feature map. The C2f module fuses shallow detail features and deep semantic features through cross-layer connections. The SPPF module aggregates the global context information of the table tennis ball juggling action;

[0037] The Neck layer network uses upsampling and Concat operations to fuse multi-scale feature maps;

[0038] The Head layer network adopts a decoupled head structure to generate classification feature maps and bounding box prediction feature maps respectively.

[0039] Preferably, the transfer learning strategy includes:

[0040] Using a pre-trained YOLOv8 model as the source model and adjusting the output dimension of its fully connected layer to match the classification categories of the table tennis ball juggling action;

[0041] Adding an RNN layer to the source model to enhance the ability to extract temporal features;

[0042] Adopting a combination of cross-entropy loss function and CIoU loss function to jointly optimize the model parameters. The formula is:

[0043] L = λ1·L cls + λ2·L CIou

[0044] where L cls is the classification loss, L CIou is the bounding box regression loss, and λ1 and λ2 are weight coefficients.

[0045] Preferably, the hitting judgment algorithm further includes:

[0046] Defining a valid hitting area, and determining an invalid hit when the center of the ball exceeds a preset proportional range of the racket bounding box;

[0047] Identifying missed balls or abnormal juggling actions through the acceleration change of the ball's movement trajectory;

[0048] Based on the Numpy module, counting the number of times the spike signal exceeds the threshold, and calculating the number of juggling times per unit time.

[0049] Preferably, the specific implementation method of the real-time feedback of the mobile terminal is:

[0050] Encapsulating an HTTP interface through FastAPI, receiving the video stream and returning the annotated result video and counting data in JSON format;

[0051] The inference results are encoded into an MP4 file in H.264 format using the imageio library, which supports real-time playback on the web page;

[0052] The comparison between the correct and incorrect actions is dynamically shown in the video in the form of animated superposition.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. By constructing a deep learning action recognition model that combines CNN and RNN, the problem that existing action recognition technologies are difficult to accurately identify and analyze complex and variable table tennis juggling actions is solved, and the technical effect of being able to accurately extract spatial features, capture the coherence and sequence of actions, and accurately identify various actions is achieved;

[0055] 2. By analyzing each frame of the uploaded video through the model and calculating various parameters in combination with pose data, the problem of not being able to know whether the force application points and swinging trajectories of teenagers' juggling in the air are reasonable and the related quantitative indicators of juggling is solved, and the technical effect of comprehensively and deeply analyzing training actions and providing rich data support for training guidance is achieved;

[0056] 3. By showing the correct demonstration of incorrect actions and giving improvement suggestions in the form of text, pictures or animations in the mobile phone applet, and generating a training report, the problem that teenagers cannot know whether their actions are standard and lack means of tracking training progress during independent practice is solved, and the technical effect of helping teenagers correct mistakes in time, understand the training situation, and improve the scientificity and effect of training is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Other features, objects and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0058] Figure 1 Schematic diagram for judging the hitting action based on the IOU value;

[0059] Figure 2 Schematic diagram of the relative change in the height of the racket and the table tennis ball;

[0060] Figure 3a Schematic diagram of the signal threshold transition of the spike counting algorithm (from static to dynamic);

[0061] Figure 3b Schematic diagram of the signal threshold transition of the spike counting algorithm (from dynamic to static)

[0062] Figure 4 Schematic diagram of the curve of the change in the training accuracy of the model;

[0063] Figure 5 Schematic diagram of the accuracy-confidence curve;

[0064] Figure 6 is a precision-recall curve graph;

[0065] Figure 7 is an F1-confidence curve graph;

[0066] Figure 8 is a flowchart of an intelligent recognition method for table tennis air-bouncing actions based on deep learning. Specific implementation manners

[0067] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all belong to the protection scope of the present invention.

[0068] Embodiment 1

[0069] All YOLO object detection models before YOLOv3 were written in C language and used the Darknet framework. Ultralytics released the first YOLO (YOLOv3) implemented using the PyTorch framework; after YOLOv3, Ultralytics released YOLOv5. In January 2023, Ultralytics released YOLOv8, which includes five models for detection, segmentation, and classification. The present invention uses the YOLOv8 detection model, and compared with the previous versions, the accuracy and inference speed have been improved to varying degrees.

[0070] The YOLOv8 network is divided into three parts, namely the backbone network, the Neck layer network, and the Head layer network. The network structure of YOLOv8 will be introduced below in these three orders. First, the network architecture of YOLOv8l will be used to illustrate.

[0071] The backbone network is used to extract features from the input image. The backbone network is mainly composed of CBS modules, C2f modules, and SPPF modules.

[0072] The C2f module realizes efficient feature fusion. In the recognition of the table tennis ball juggling action in the air, it can efficiently fuse the feature maps from different convolutional layers, combining the low-level features with rich details in the shallow layer and the high-level features containing semantic information in the deep layer. This ensures that the model can not only pay attention to the subtle limb changes in the juggling action but also grasp the overall action pattern, improving the comprehensive extraction ability of complex juggling action features, and thus providing more comprehensive and rich feature information for subsequent accurate action recognition. In addition, the C2f module can also enhance the robustness of the model. There are individual differences in the table tennis ball juggling action, and the juggling speed, strength, and posture of different teenagers may vary. By fusing multi-scale features, the model has stronger adaptability to these differences.

[0073] The SPPF module can aggregate the global context information of the table tennis ball juggling action in a short time. During the recognition process, it quickly collects information from different regions of the image through parallel max-pooling operations at different scales. This helps the model understand the overall process and situation of the juggling action from a macroscopic perspective, improving the analysis ability of the coherence and sequence of continuous juggling actions. As a result, it can more accurately identify the juggling rhythm, whether there are missed balls, etc., obtain key global information to improve the recognition performance, and ensure the efficiency of the recognition process, without causing lags or delays due to excessive computational burden, providing a smooth and real-time action analysis and feedback experience.

[0074] The CBS module represents Conv (Convolution, Convolutional Neural Network), BN (Batch Normalization, Normalization), and SILU (Sigmoid Linear Unit, an adaptive activation function), and is used to perform convolution on the input image or feature map. The result of the convolution is a decrease in resolution and an increase in the number of channels. There are a total of 5 CBS modules. The resolution of the input image changes from 640*640 to 20*20, and the number of input channels changes from 3 to 512.

[0075] The Neck layer network is used to perform feature fusion on the feature maps output by the backbone network, that is, to Concat (Concatenate, when constructing a neural network model, the Concat operation is used to connect multiple tensors along the specified dimension) the features extracted at these three different scales through upsampling. After two upsamplings, the first feature map (80*80) is output to the Head layer network, and then the feature map is downsampled through the CBS module to output the remaining two feature maps (40*40 and 20*20) to the Head layer network.

[0076] The Head layer network designs and generates specific feature maps according to the number of categories. YOLOv8 uses a decoupled head method to generate them. Decoupling means generating feature maps for predicting Cls (Classification) and Box (BoundingBox) separately. For example, for the input feature map P3 with a scale of (1, 256, 80, 80), the branch for generating the Box prediction feature map will generate a feature map of (1, 64, 80, 80) through two CBS modules and a Cov2d (Convolution2D, a common operation in convolutional neural networks in deep learning, specifically for two-dimensional data such as images). The branch for generating the Cls prediction feature map will generate a feature map of (1, nc, 80, 80) through two CBS modules and a Cov2d, where nc represents the predicted categories. After passing through the Head layer, 3 feature maps for predicting Boxes (1, 64, 80, 80), (1, 64, 40, 40), and (1, 64, 20, 20) will be generated; and 3 feature maps for predicting Cls (1, nc, 80, 80), (1, nc, 40, 40), and (1, nc, 20, 20) will be generated.

[0077]

[0078] Figure Comparison of YOLOv8 Model Results at Different Sizes

[0079] Transfer Learning

[0080] Transfer learning is a method that uses the knowledge learned from a task to improve the learning of a new task. It can reduce data requirements, training time, generalization error, and the computational cost of building a model on a new task. Currently, transfer learning mainly includes the following three methods.

[0081] (1) Full Weight Transfer. Directly transfer all the weights of the model trained on the source task to the target task. This method assumes that the source task and the target task have similar feature representations and output spaces, and is applicable to cases where the similarity between the source task and the target task is relatively high.

[0082] First, select a model pre-trained on large-scale visual tasks. Such models have learned from a large amount of image data. In this patent, the Yolo pre-trained model is used as the source model. At the same time, collect video data of table tennis players' actions of hitting the ball into the air, covering scenes of hitting the ball by different teenagers, various shooting angles, and various lighting and background environments. Preprocess these videos (extract key frame images, etc.) and label the data to form a target task dataset.

[0083] Then, according to the requirements of the target task, the source model structure is appropriately adjusted. Since the output dimension of the fully connected layer of the source model does not match the number of categories in the table tennis ball juggling action classification, the structure of the fully connected layer needs to be modified. Because the target model also focuses on the time series characteristics of the action, a recurrent neural network (RNN) layer is added.

[0084] Finally, in the way of the weights of the Conv2d layer, the parameters of the BN layer, etc., the weights and parameters of the corresponding layers of the source model are copied to the corresponding layers of the target model to complete the weight initialization, enabling the target model to utilize the general features learned by the source model. At the same time, according to the characteristics of the table tennis ball juggling action recognition task, a cross-entropy loss function is defined to implement a multi-classification task (such as normal juggling, incorrect juggling, missed ball, etc. classification). Using the target task dataset, the gradient of the loss function with respect to the model weights is calculated through the backpropagation algorithm. During the fine-tuning process, the model gradually adapts to the specific patterns of the table tennis ball juggling action and optimizes its performance. The formula is as follows:

[0085]

[0086] (2) Feature extraction. The underlying feature extraction part of the model trained on the source task is used as a fixed feature extractor, and only the classifier part of the target task is trained. In this way, the general features learned on the source task can be used to extract the feature representation of the target task, thereby accelerating the training speed of the target task and improving performance. According to the characteristics and requirements of the table tennis ball juggling action recognition task, a classifier structure is designed to classify actions such as normal juggling, incorrect juggling (such as incorrect hitting posture, unstable juggling rhythm, etc.), and missed ball. The parameters of the constructed classifier part are initialized to enable the classifier to converge faster and learn better.

[0087] (3) Fine-tuning. On the basis of feature extraction, some or all of the weights of the source task are fine-tuned to adapt to the specific requirements of the target task. Fine-tuning can adjust the model parameters by performing limited training on the target task to make it better adapt to the data distribution and feature representation of the target task. Using the previously collected and preprocessed video data of table tennis ball juggling actions against the air, the training set, validation set, and test set are divided. The training set data is input into the target model for training. During the training process, the training data is sequentially input into the target model. The data first passes through the fixed underlying feature extractor of the source model to extract features, and then these features are input into the classifier for classification prediction. The loss between the prediction result and the true label is calculated, the gradient of the loss with respect to the classifier parameters is calculated through the backpropagation algorithm, and the classifier parameters are updated using the optimizer, while the parameters of the underlying feature extractor of the source model remain unchanged, enabling the classifier to gradually adapt to the classification task of the table tennis ball juggling action and optimize the classification performance.

[0088] First, perform data normalization on the training set, validation set, and test set data, scaling the image pixel values to a specific range.

[0089] Then, in the fine-tuning stage, restart the training loop. In each round of training, input the training data into the target model. The data first passes through the feature extractor (possibly with some layers unfrozen) to extract features, and then is input into the classifier for classification prediction. Calculate the loss between the prediction result and the true label, calculate the gradients of the loss with respect to the model parameters (including the parameters of the unfrozen layers and the classifier parameters) through the backpropagation algorithm, and use the optimizer to update the parameters. After each round of training, use the validation set data to validate the model and calculate the loss and evaluation metrics (such as accuracy, recall, etc.) on the validation set. If the performance on the validation set does not improve within a certain number of rounds (5 rounds are used), that is, adopt the early stopping strategy to stop training to prevent overfitting of the model.

[0090] After fine-tuning, use the test set data to perform the final evaluation of the model, calculate metrics such as accuracy, recall, and F1-score, and comprehensively evaluate the performance of the model on the task of table tennis ball juggling action recognition.

[0091] Evaluation Metrics

[0092] In the object detection task, precision, recall, and F1-score are three important evaluation metrics, and they have the following physical meanings respectively:

[0093] Precision represents the proportion of samples that are actually of a certain class among all samples predicted by the model as that class. A high precision means that the model's prediction for that class is relatively accurate and there are fewer false positives. The formula is:

[0094]

[0095] where TP is the number of samples correctly predicted as that class, and FP is the number of samples wrongly predicted as that class.

[0096] Recall represents the proportion of samples that are actually of a certain class among all samples that are correctly predicted by the model as that class. A high recall means that the model has a strong ability to capture samples of that class and there are fewer false negatives. The formula is:

[0097]

[0098] where FN is the number of samples that are actually of that class but are wrongly predicted as other classes.

[0099] The F1 value is the harmonic mean of precision and recall, which is used to balance between precision and recall. The higher the F1 value, the better the balance achieved by the model between precision and recall, making it suitable for situations with imbalanced sample distributions. The formula is:

[0100]

[0101] Table Tennis Algorithm Model

[0102] The premise of table tennis counting is the accurate recognition of table tennis balls, which belongs to the category of object detection. On the basis of accurate recognition, it is necessary to comprehensively calculate the position changes before and after the ball and the racket to achieve an accurate judgment of hitting the ball. In this invention, the YOLOv8 algorithm is used for object recognition, and after recognition, a self-developed comprehensive algorithm is used for action judgment. Under the framework of YOLOv8, for this table tennis ball juggling algorithm, the core expressions mainly revolve around the detection and classification of objects. For class prediction, the model outputs a class probability vector P, whose dimension is the vector C (C is the number of target classes, in the table tennis ball juggling scenario, it is classes such as table tennis balls, rackets, and players):

[0103] P = (p1, p2, p3..., p c )

[0104] During the training process, a loss function is used to optimize the model parameters. According to the CIoU loss, for each predicted bounding box and the ground truth bounding box, the CIoU loss L CIoU has the following formula:

[0105]

[0106] where IoU is the intersection over union of the predicted bounding box and the ground truth bounding box, is the square of the Euclidean distance between the centers of the ground truth bounding box and the predicted bounding box, c is the diagonal length of the smallest enclosing rectangle containing the ground truth bounding box and the predicted bounding box, is the balance parameter, ν is an index to measure the aspect ratio consistency between the ground truth bounding box and the predicted bounding box, and by minimizing L CIoU and the cross-entropy loss L CE for class prediction to update the model parameters θ, the formula is as follows:

[0107]

[0108] where N is the number of training samples, and λ is the weight to balance the CIoU loss and the cross-entropy loss.

[0109] Such as Figure 1, in the initial solution, the method of using the IOU value between the board and the ball was adopted to judge the hitting action. When the ball left the hitting area, since the two detection frames did not intersect, the IOU remained 0 continuously. When the ball entered the board area, the two detection frames had an intersecting area, and as the hitting action continued, a spike would appear in the IOU value. By judging this spike, the hitting action could be determined.

[0110] However, with the in-depth research, the following problems were found. First, the recognition algorithm had relatively high requirements for the position of the camera. It had to be shot at an oblique downward angle. Otherwise, when the board was horizontal, it occupied a relatively small proportion in the picture, and it was very easy to lose the object, resulting in the inability to complete the calculation. Second, the ball speed was relatively fast at the moment of hitting, and the frame rate requirement was relatively high. The IOU calculation needed to accurately detect the ball and the board. In order to improve the detection accuracy, the camera had to be accurately aligned with the racket. However, in the actual examination room, the field of view of the camera was relatively large, and the racket would go out of the picture.

[0111] Considering that the camera angle was relatively fixed when the athlete hit the ball, based on the accurate recognition of the racket and the ball, a comprehensive judgment was made according to the relative position relationship between the ball and the racket. During the hitting process, the relative change in the height of the racket and the ball was as Figure 2 .

[0112] From Figure 2 , it was not difficult to find that the hitting action was a very regular action. Whether it was the ball or the racket, it had relatively stable frequency and amplitude. During the normal hitting process, the following several constraint conditions would obviously hold. First, the ball was in the upper area of the racket and could not be in the lower area. Second, the left and right areas of the ball would not vary greatly. If the abscissa value deviated beyond a certain range, it could be considered that the ball had got out of control and there was no need to continue counting. Third, the size of the ball and the horizontal size of the racket were relatively fixed, while the vertical size of the racket would be large or small due to the angle. Based on this, a hitting judgment algorithm under multiple constraint conditions was designed.

[0113] The main process is as follows:

[0114] a) Judge whether the racket and the ball appear simultaneously;

[0115] b) Set the ROI (Region of Interest) of the picture proportionally. It is invalid outside the region. Judge that if the left edge of the target is on the left side of the effective region A, the right edge is on the right side of the effective region A, or the upper edge is on the upper side of the effective region, and the lower edge is on the lower side of the effective region, then it is invalid;

[0116] c) Similarly, judge whether the ball is near the racket. If it is not within the range, it is invalid;

[0117] d) When the ball goes out of the range or other invalid conditions are met, immediately calculate the number of hits during this period of valid time;

[0118] e) Add up the numbers within each effective hitting time period, and the sum is the total number.

[0119] Among them, the peak calculation value uses the reciprocal of the sum of the vertical distance between the ball and the racket and the racket, that is, 1 / (dis_y + 1). In this way, when the ball is closest to the racket, its value is the largest. The peak counting uses the sum algorithm of the Numpy module, such as Figure 3a and Figure 3b . This algorithm determines the number of pulses by judging the relationship between the signal value and the threshold, that is, by counting the number of times the signal jumps from less than the threshold to greater than or equal to the threshold. The number of times changing from below a certain threshold (which may represent a stationary or non-bouncing state) to above or equal to the threshold (representing the occurrence of a bouncing action) is used to obtain the bounce count, providing a key feature of the bounce count for subsequent training models. The trained model can accurately identify small and fast-moving objects (table tennis balls), table tennis rackets, etc. in pictures. And the number of bounces can be analyzed based on the time series.

[0120] Dataset and Parameter Settings

[0121] To verify the effectiveness of the algorithm, a self-made Pingpong2k dataset is used in the experiment, with each image containing at least one of the two types of objects: the racket and the ball. The training set, validation set, and test set are divided in a ratio of 7:2:1, and the size of the dataset is enhanced.

[0122] The experimental environment uses the Windows 11 operating system, with 32GB of running memory, an Intel Core I5 8400 processor, an A2000 GPU, an NVIDIA GTX 3060 GPU, the Python 3.11 programming language, and the model is implemented based on the PyTorch deep learning framework.

[0123] Some learning parameters are as follows:

[0124] dropout: 0.0

[0125] warmup_epochs: 3.0

[0126] iou: 0.7

[0127] mosaic: 1.0

[0128] lr0: 0.01

[0129] lrf: 0.01

[0130] weight_decay: 0.0005

[0131] A total of 100 epochs are trained. The training results are as Figures 4 to 7 .

[0132] As Figure 4 , at the initial state, the abscissa starts from 0. In the early stage (the first few training steps), the accuracy value of the model fluctuates greatly, with the lowest being about 0.5 and the highest approaching 0.7. This indicates that in the initial stage of training, the model has not fully learned, the recognition of relevant features of table tennis ball juggling is unstable, and it is in the stage of exploring and adapting to the data. As the number of training rounds increases, the accuracy value generally shows an upward trend and finally stabilizes at about 0.85. The model gradually learns effective features, the recognition ability of table tennis ball juggling is continuously improved, and it tends to be stable in the later stage.

[0133] As Figure 5 , it is a Precision-Confidence Curve graph in the training of the table tennis ball juggling model. The annotation "all classes 1.0 at 0.808" in the graph means that when the confidence reaches 0.808, the accuracy of all classes can reach 1.0. This means that at a relatively high confidence threshold, the model can achieve very high accuracy in detecting all classes, that is, there are almost no misdetection cases.

[0134] As Figure 6 , it is a Precision-Recall Curve graph in the training of the table tennis ball juggling model, which reflects the performance of the model when detecting table tennis balls (ball) and rackets (bat), as well as the comprehensive performance of all classes. The mean average precision (mAP@0.5) of all classes in the graph is 0.865. The curve trend synthesizes the situations of table tennis balls and rackets, reflecting the performance of the model in the overall object detection task.

[0135] As Figure 7 , it is an F1-Confidence Curve graph, which is used to evaluate the performance of the table tennis ball juggling model when detecting table tennis balls (ball), rackets (bat), and all classes. The F1 value of all classes is 0.83, and the corresponding confidence is 0.115. This curve comprehensively reflects the overall detection performance of the model for table tennis balls and rackets. When the confidence is 0.115, the F1 value can reach 0.83, indicating that the model can better balance accuracy and recall as a whole at a relatively low confidence, but as the confidence increases, the F1 value will also decrease.

[0136] As Figure 8, is a flow chart of an intelligent recognition method for table tennis air-tossing actions based on deep learning, including the following steps: Step 1: Collect real-time video data and attitude sensor data of air-tossing the table tennis ball through a mobile terminal; Step 2: Build a deep learning action recognition model, which consists of a CNN module, an RNN module, and a YOLO target detection module. Among them: The CNN module is used to extract the spatial features in the video frame, including the position information of the racket, the table tennis ball, and the human body key points; The RNN module is used to capture the temporal correlation of actions in consecutive video frames; The YOLO target detection module is based on the YOLOv8 architecture, realizes multi-scale feature fusion through the C2f module, and aggregates global context information through the SPPF module; Step 3: Optimize the pre-trained YOLO model using a transfer learning strategy, including weight transfer, fixing the feature extractor, and fine-tuning the classifier; Step 4: According to the relative position relationship between the racket and the table tennis ball, design a hitting judgment algorithm with multiple constraints, including: determining the hitting action through the peak detection of the vertical distance between the ball and the racket; combining the effectiveness judgment of the ROI area of the racket in the picture and the analysis of the hitting frequency; Step 5: Real-time feedback the recognition results through the mobile terminal, including the demonstration of correcting wrong actions, the generation of training reports, and the display of quantitative indicators.

[0137] Interface integration

[0138] Use the Fastapi package to make the required interfaces into standard http interfaces for easy front-end calling. After receiving table tennis and football videos, send them to the inference module for inference calculation, detect frame by frame and draw the results on the original image. The http interface returns the counting results and the address of the result video. The server will re-encapsulate the results into an MP4 video encoded in h264 for direct playback on the web page. At first, the cv2 package was used for encoding, and it was found that h264 encoding was not supported in the linux environment and needed to be compiled, but finally there were still problems with the environment. Finally, the imageio package was used.

[0139] Embodiment 2

[0140] The present invention also provides an intelligent recognition system for table tennis air-tossing actions based on deep learning. The intelligent recognition system for table tennis air-tossing actions based on deep learning can be realized by executing the flow steps of the intelligent recognition method for table tennis air-tossing actions based on deep learning. That is, those skilled in the art can understand the intelligent recognition method for table tennis air-tossing actions based on deep learning as the preferred implementation manner of the intelligent recognition system for table tennis air-tossing actions based on deep learning.

[0141] According to the intelligent recognition system for table tennis air-bouncing actions based on deep learning provided by the present invention, it includes: Module M1: Collect real-time video data and attitude sensor data of air-bouncing the table tennis ball through a mobile terminal; Module M2: Construct a deep learning action recognition model, which consists of a CNN module, an RNN module, and a YOLO target detection module. Among them: The CNN module is used to extract spatial features in video frames, including the position information of the racket, the table tennis ball, and human key points; The RNN module is used to capture the temporal correlation of actions in consecutive video frames; The YOLO target detection module is based on the YOLOv8 architecture, realizes multi-scale feature fusion through the C2f module, and aggregates global context information through the SPPF module; Module M3: Optimize the pre-trained YOLO model using a transfer learning strategy, including weight transfer, fixing the feature extractor, and fine-tuning the classifier; Module M4: Design a hitting judgment algorithm with multiple constraint conditions according to the relative position relationship between the racket and the table tennis ball, including: Determining the hitting action through peak detection of the vertical distance between the ball and the racket; Combining the effectiveness judgment of the ROI area of the racket in the picture and the analysis of the hitting frequency; Module M5: Real-time feedback the recognition results through the mobile terminal, including demonstration of correcting wrong actions, generation of training reports, and display of quantitative indicators.

[0142] The CNN module includes: A backbone network, which consists of a CBS module, a C2f module, and an SPPF module. Among them, the CBS module is used to perform convolution on the input image or feature map. The C2f module fuses shallow detail features and deep semantic features through cross-layer connections. The SPPF module aggregates the global context information of the table tennis ball bouncing action; A Neck layer network, which fuses multi-scale feature maps through upsampling and Concat operations; A Head layer network, which adopts a decoupled head structure to generate a classification feature map and a bounding box prediction feature map respectively.

[0143] The transfer learning strategy includes: Using the pre-trained YOLOv8 model as the source model, and adjusting the output dimension of its fully connected layer to match the classification categories of table tennis ball bouncing actions; Adding an RNN layer to the source model to enhance the ability to extract temporal features; Jointly optimizing the model parameters using the cross-entropy loss function and the CIoU loss function. The formula is: L = λ1·L cls +λ2·L CIou , where L cls is the classification loss, L CIou is the bounding box regression loss, and λ1, λ2 are weight coefficients.

[0144] The hitting judgment algorithm further includes: defining a valid hitting area, and determining an invalid hit when the center of the sphere exceeds a preset proportional range of the racket bounding box; identifying a missed ball or an abnormal ball juggling action through the acceleration change of the sphere's movement trajectory; and counting the number of times the spike signal exceeds the threshold based on the Numpy module to calculate the number of ball juggling times per unit time.

[0145] The specific implementation of the real-time feedback of the mobile terminal is as follows: encapsulating the HTTP interface through FastAPI, receiving the video stream and returning the annotated result video and the counting data in JSON format; encoding the inference result into an MP4 file in H.264 format using the imageio library to support real-time playback on the web page; and dynamically displaying the comparison between the correct action and the wrong action in the video in the form of an animated overlay.

[0146] Those skilled in the art know that in addition to implementing the system, device, and their respective modules provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system, device, and their respective modules provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc., to achieve the same program. Therefore, the system, device, and their respective modules provided by the present invention can be regarded as a hardware component, and the modules included therein for implementing various programs can also be regarded as the structure within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing the method or the structure within the hardware component.

[0147] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined arbitrarily with each other.

Claims

1. An intelligent recognition method for the action of table tennis ball juggling in the air based on deep learning, characterized in that, Including: Step 1: Collect real-time video data and attitude sensor data of table tennis juggling in the air through a mobile terminal; Step 2: Construct a deep learning action recognition model, which consists of a CNN module, an RNN module, and a YOLO object detection module. Among them: the CNN module is used to extract spatial features in video frames, including the position information of the racket, table tennis ball, and human key points; the RNN module is used to capture the temporal correlation of actions in consecutive video frames; the YOLO object detection module is based on the YOLOv8 architecture, realizes multi-scale feature fusion through the C2f module, and aggregates global context information through the SPPF module; Step 3: Optimize the pre-trained YOLO model using transfer learning strategies, including weight transfer, fixing the feature extractor, and fine-tuning the classifier; Step 4: Design a hitting judgment algorithm with multiple constraint conditions according to the relative position relationship between the racket and the table tennis ball, including: determining the hitting action through the peak detection of the vertical distance between the ball and the racket; combining the validity judgment of the ROI area of the racket in the picture and the hitting frequency analysis; Step 5: Real-time feedback the recognition results through the mobile terminal, including error action correction demonstration, training report generation, and display of quantitative indicators.

2. The intelligent recognition method for the table tennis empty toss action based on deep learning according to claim 1, characterized in that The CNN module includes: The backbone network, which is composed of CBS modules, C2f modules, and SPPF modules. Among them, the CBS module is used to perform convolution on the input image or feature map, the C2f module fuses shallow detail features and deep semantic features through cross-layer connections, and the SPPF module aggregates the global context information of the table tennis juggling action; The Neck layer network, which fuses multi-scale feature maps using upsampling and Concat operations; The Head layer network, which adopts a decoupled head structure to generate classification feature maps and bounding box prediction feature maps respectively.

3. The intelligent recognition method for the table tennis ball tossing action into the air based on deep learning according to claim 1, characterized in that, The transfer learning strategy includes: Using the pre-trained YOLOv8 model as the source model and adjusting the output dimension of its fully connected layer to match the classification categories of table tennis juggling actions; Adding an RNN layer to the source model to enhance the ability to extract temporal features; Jointly optimizing the model parameters using the cross-entropy loss function and the CIoU loss function, and the formula is: L = λ1·L cls + λ2·L CIou Among them, L cls is the classification loss, and L CIou is the bounding box regression loss. λ1 and λ2 are weight coefficients.

4. The intelligent recognition method for the table tennis air-tossing action based on deep learning according to claim 1, wherein The hitting judgment algorithm further includes: Defining a valid hitting area, and determining an invalid hit when the center of the sphere exceeds a preset proportional range of the racket bounding box; Identifying missed balls or abnormal juggling actions through the acceleration change of the sphere's movement trajectory; Based on the Numpy module, counting the number of times the spike signal exceeds the threshold, and calculating the number of juggling times per unit time.

5. The intelligent recognition method for the table tennis empty toss action based on deep learning according to claim 1, characterized in that The specific implementation method of the real-time feedback of the mobile terminal is: Encapsulating the HTTP interface through FastAPI, receiving the video stream and returning the annotated result video and counting data in JSON format; Using the imageio library to encode the inference result into an MP4 file in H.264 format, supporting real-time playback on the web page; Dynamically displaying the comparison between correct actions and incorrect actions in the video in the form of animated superposition.

6. An intelligent recognition system for table tennis ball tossing actions against the air based on deep learning, characterized in that, Including: Module M1: Collect real-time video data and attitude sensor data of table tennis juggling in the air through a mobile terminal; Module M2: Build a deep learning action recognition model, which consists of a CNN module, an RNN module, and a YOLO object detection module. Among them: The CNN module is used to extract spatial features in video frames, including the position information of the racket, table tennis ball, and human key points; The RNN module is used to capture the temporal correlation of actions in consecutive video frames; The YOLO object detection module is based on the YOLOv8 architecture, realizes multi-scale feature fusion through the C2f module, and aggregates global context information through the SPPF module; Module M3: Optimize the pre-trained YOLO model using transfer learning strategies, including weight transfer, fixing the feature extractor, and fine-tuning the classifier; Module M4: Design a hitting judgment algorithm with multiple constraints according to the relative position relationship between the racket and the table tennis ball, including: judging the hitting action through the peak detection of the vertical distance between the ball and the racket; combining the validity judgment of the ROI area of the racket in the picture and the hitting frequency analysis; Module M5: Real-time feedback the recognition results through the mobile terminal, including the demonstration of correcting wrong actions, the generation of training reports, and the display of quantitative indicators.

7. The intelligent recognition system for the table tennis ball-tossing action against the air based on deep learning according to claim 6, characterized in that, The CNN module includes: The backbone network is composed of CBS modules, C2f modules, and SPPF modules. Among them, the CBS module is used to perform convolution on the input image or feature map. The C2f module fuses shallow detail features and deep semantic features through cross-layer connections. The SPPF module aggregates the global context information of the table tennis ball juggling action; The Neck layer network uses upsampling and Concat operations to fuse multi-scale feature maps; The Head layer network adopts a decoupled head structure to generate classification feature maps and bounding box prediction feature maps respectively.

8. The intelligent recognition system for table tennis air-bouncing actions based on deep learning according to claim 6, characterized in that, The transfer learning strategy includes: Use the pre-trained YOLOv8 model as the source model and adjust the output dimension of its fully connected layer to match the classification categories of table tennis ball juggling actions; Add an RNN layer to the source model to enhance the ability to extract temporal features; Adopt the cross-entropy loss function and the CIoU loss function to jointly optimize the model parameters. The formula is: L = λ1·L cls + λ2·L CIou Among them, L cls is the classification loss, and L CIou is the bounding box regression loss. λ1 and λ2 are weight coefficients.

9. The intelligent recognition system for table tennis air-tossing actions based on deep learning according to claim 6, characterized in that, The hitting judgment algorithm further includes: Define the effective hitting area, and judge an invalid hit when the center of the ball exceeds the preset proportional range of the racket bounding box; Identify missed balls or abnormal juggling actions through the acceleration change of the ball's movement trajectory; Based on the Numpy module, count the number of times the spike signal exceeds the threshold, and calculate the number of juggling times per unit time.

10. The intelligent recognition system for table tennis ball tossing action in the air based on deep learning according to claim 6, characterized in that, The specific implementation method of the real-time feedback of the mobile terminal is: Package the HTTP interface through FastAPI, receive the video stream and return the annotated result video and the counting data in JSON format; Use the imageio library to encode the inference result into an MP4 file in H.264 format, supporting real-time playback on the web page; Dynamically display the comparison between the correct action and the wrong action in the video in the form of an animated overlay.

Images