Multi-task data real-time detection method and device for table tennis game videos
By using a deep neural network with three-dimensional convolutional layers and Transformer self-attention mechanism in table tennis match videos, a real-time detection model for multi-task data was constructed. This solved the problem of low accuracy in ball detection and hitting action recognition in table tennis match videos, and achieved real-time and accurate multi-task detection.
Patent Information
- Application Number
- CN202310733836.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-20
- Publication Date
- 2025-09-19
- Estimated Expiration
- 2043-06-20
AI Technical Summary
Existing computer vision methods have difficulty achieving real-time and accurate ball detection and batting action recognition in table tennis game videos, especially because table tennis balls are small and move at high speeds, resulting in low detection and recognition accuracy of existing models.
A deep neural network with three-dimensional convolutional layers is used to extract frame-by-frame visual features. Combined with the Transformer self-attention mechanism, a real-time detection model for multi-task data is constructed. The model is optimized through unsupervised pre-training and supervised training to adapt to table tennis game scenarios.
The accuracy of ball detection and hitting action recognition is improved, real-time, end-to-end multi-task detection of table tennis game videos is achieved, and processing accuracy and speed are improved.
Smart Images
Figure CN116958861B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of sports video data mining, and in particular relates to a multi-task data real-time detection method and device for table tennis match videos. Background Art
[0002] Visual analytics, a product of data visualization, aims to facilitate analytical reasoning and gain data insights through interactive visual interfaces and computer vision technologies. With the rapid development of computer science and technology, visual analytics, as an effective data analysis tool, is increasingly being applied to sports data analysis. It not only provides guidance and assistance to coaches and athletes during training and competition, but also enhances the effectiveness of various media communications. In the field of professional competition data analysis, visual analytics technology can more directly and effectively demonstrate the temporal and spatial characteristics of a match and the technical and tactical characteristics of athletes, attracting the attention of numerous professional sports data analysts.
[0003] For table tennis match videos, ball detection and batting motion recognition are two of the most critical elements in visual sports data analysis. Accurate ball recognition not only enables better visualization of match videos, such as showing the ball's trajectory, but also assists with downstream data mining tasks, such as batting motion classification. Batting motion recognition is key to round segmentation and tactical analysis, and can also provide statistical data for multiple rounds. Based on multi-round batting motion data, players' ball paths and tactics can be accurately and efficiently analyzed. This not only provides reliable support for national table tennis team athletes to study opponents, refine tactics, and effectively prepare for matches, but also provides amateur enthusiasts with a cost-effective "professional data analyst" to improve ball paths and research tactics. Therefore, it is necessary to develop more intelligent data mining methods for table tennis match videos to make real-time ball detection and batting motion recognition in table tennis match videos more efficient and accurate.
[0004] Computer vision methods are increasingly being applied to data extraction for visual analysis in sports videos. However, using deep neural network models to achieve real-time, end-to-end ball detection and batting motion recognition in live table tennis match videos presents numerous challenges. First, because the ping-pong balls in table tennis videos are not only tiny but also move at high speeds, they often appear as blurred, elongated shadows, posing new challenges compared to traditional detection and recognition tasks. Second, the batting motions of players in table tennis videos are often subtle, and high-speed motion can easily cause image blur. Therefore, extracting generalizable visual features is crucial for multi-task, end-to-end models.
[0005] To address the challenges of ball detection and batting action recognition in live table tennis match videos, existing approaches employ the table tennis real-time spatiotemporal video analysis network (TTNet). This network extracts video features using a two-dimensional convolutional neural network (2D convolutional neural network) for semantic segmentation, two-stage ball detection, and event detection. However, in testing, the accuracy of both ball location and batting event detection is too low. Commonly used video feature extraction methods include the I3D network and the Contrastive Action Representation Learning (CARL) framework. I3D is a two-stream convolutional neural network that takes video and optical flow as input and outputs video categories for feature modeling. However, I3D severely downsamples the video in the spatial dimension, losing a significant amount of low-dimensional visual feature information. This makes it incapable of detecting subtle objects like table tennis balls. Furthermore, I3D requires optical flow extraction, hindering real-time performance. CARL uses ResNet and Transformer as its backbone to extract event features from videos and utilizes contrastive learning for training on unsupervised data. However, CARL primarily focuses on video feature extraction and is incapable of ball detection and batting action recognition. Summary of the Invention
[0006] In view of the above, the purpose of the present invention is to provide a multi-task data real-time detection method and device for table tennis game videos, which adopts computer vision methods to extract frame-by-frame visual features through a deep neural network with a three-dimensional convolutional layer, and realizes multi-task data visualization and real-time detection of ball detection and hitting action recognition based on the extracted frame-by-frame visual features.
[0007] To achieve the above-mentioned purpose, the present invention provides the following technical solutions:
[0008] A multi-task data real-time detection method for table tennis game videos includes the following steps:
[0009] Build a multi-task data real-time detection model, including a video feature extraction module, a ball detection module, and a batting action recognition module;
[0010] Unsupervised pre-training of the multi-task data real-time detection model is performed to obtain pre-training parameters. Based on the pre-training parameters, supervised training is performed using a labeled table tennis game video dataset.
[0011] Utilize a deep neural network with a 3D convolutional layer added to the trained video feature extraction module to extract frame-by-frame visual features of table tennis match videos;
[0012] The trained ball detection module is used to convert the frame-by-frame visual features into a single-channel confidence heat map to determine the frame-by-frame position of the table tennis ball.
[0013] The trained batting action recognition module is used to reduce the dimensionality of the frame-by-frame visual features and perform position encoding. After further dimensionality reduction, linear regression is used to obtain the player's batting confidence frame by frame to determine the batting time and number of batting times.
[0014] The output results of the ball detection module and the batting action recognition module are visualized using drawing tools and annotated in the table tennis game video, realizing real-time visualization of multi-task data for ball detection and batting action recognition.
[0015] The video feature extraction module includes a three-dimensional convolution layer, a maximum pooling layer and multiple convolution layers connected in sequence.
[0016] The ball detection module includes multiple convolutional layers, upsampling layers and convolutional layers connected in sequence. The multiple convolutional layers of the ball detection module are jump-connected with the maximum pooling layer and multiple convolutional layers in the video feature extraction module.
[0017] The batting action recognition module includes a maximum pooling + downsampling layer, a Transformer encoder and a linear layer connected in sequence.
[0018] The unsupervised pre-training of the multi-task data real-time detection model to obtain pre-training parameters includes: unsupervised pre-training of the video feature extraction module using a contrastive learning method to learn basic visual perception capabilities so that the frame-by-frame visual features extracted by the video feature extraction module have multi-task generalization, thereby obtaining pre-training parameters;
[0019] The supervised training based on pre-training parameters and using a labeled table tennis match video dataset includes: supervised training of a multi-task data real-time detection model using the labeled table tennis match video dataset based on the pre-training parameters, so that the model is further adapted to the task scenario of a live table tennis match video, and is more sensitive to the fast-moving table tennis ball and the hidden hitting movements of the players in the table tennis match; a binary cross-entropy loss function is used to add the loss functions of the two tasks of ball detection and hitting movement recognition to obtain a loss function in a supervised training stage, and the multi-task data real-time detection model is optimized.
[0020] The method uses a deep neural network with a three-dimensional convolutional layer added to a trained video feature extraction module to extract frame-by-frame visual features of table tennis match videos. The method includes: for the RGB three-channel input of each frame of the video, first using a three-dimensional convolutional layer to downsample the time and space dimensions to obtain a feature vector, then further downsampling through a maximum pooling layer, and finally extracting frame-by-frame visual features through multiple convolutional layers.
[0021] The method utilizes a trained ball detection module to convert frame-by-frame visual features into a heat map of single-channel confidence, thereby determining the frame-by-frame position of the table tennis ball. The method includes: receiving intermediate outputs of the maximum pooling layer and multiple convolutional layers in the video feature extraction module through multiple convolutional layers of the ball detection module, and then performing upsampling and convolution operations in the spatial dimension through an upsampling layer and a convolutional layer. The frame-by-frame visual features output by the video feature extraction module are converted into a heat map of the confidence of table tennis ball detection at various positions in the single-channel screen, thereby determining the frame-by-frame position of the table tennis ball as the ball detection result.
[0022] The method utilizes a trained batting action recognition module to reduce the dimensionality of frame-by-frame visual features and then perform position encoding. After further dimensionality reduction, linear regression is performed to obtain the player's batting confidence frame-by-frame, thereby determining the batting time and the number of batting times. The method includes: downsampling the frame-by-frame visual features output by the video feature extraction module through a maximum pooling + downsampling layer to output low-dimensional features, using a Transformer encoder to perform position encoding on the low-dimensional features, downsampling the encoded low-dimensional features through a linear layer, reducing the dimensionality again, and performing linear regression to obtain the player's batting confidence frame-by-frame, and reflecting the probability of a batting event occurring in each frame through the confidence, thereby determining the batting time and the number of batting times.
[0023] To achieve the above-mentioned object of the invention, an embodiment of the present invention further provides a multi-task data real-time detection device for table tennis match videos, comprising a model building unit, a model training unit, a video feature extraction unit, a ball detection unit, a hitting action recognition unit, and a visualization generation unit;
[0024] The model building unit is used to build a multi-task data real-time detection model, including a video feature extraction module, a ball detection module and a hitting action recognition module;
[0025] The model training unit is used to perform unsupervised pre-training on the multi-task data real-time detection model to obtain pre-training parameters, and perform supervised training based on the pre-training parameters and using the labeled table tennis game video dataset;
[0026] The video feature extraction unit is used to extract frame-by-frame visual features of the table tennis game video using a deep neural network with a three-dimensional convolutional layer added to the trained video feature extraction module;
[0027] The ball detection unit is used to convert the frame-by-frame visual features into a single-channel confidence heat map using the trained ball detection module to determine the frame-by-frame position of the table tennis ball;
[0028] The batting action recognition unit is used to use the trained batting action recognition module to reduce the dimension of the visual features frame by frame and then perform position encoding. After further dimensionality reduction, linear regression is performed to obtain the player's batting confidence frame by frame, thereby determining the batting time and number of batting times.
[0029] The visualization generation unit is used to visualize the output results of the ball detection module and the hitting action recognition module through a drawing tool, and mark them in the table tennis game video, thereby realizing multi-task data visualization and real-time detection of ball detection and hitting action recognition.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] (1) Based on the extraction of video features by a deep neural network with a three-dimensional convolutional layer, the frame-by-frame video features of table tennis game videos in the time and space dimensions can be efficiently extracted for detailed modeling, thereby improving the accuracy of ball detection and batting action recognition.
[0032] (2) The Transformer self-attention mechanism model is introduced, which has strong modeling capabilities for long sequences and can enhance the recognition of batting action events.
[0033] (3) The method of the present invention is improved based on the computer vision method, which improves the processing accuracy and speed of the model and realizes real-time, end-to-end ball detection and hitting action recognition multi-task detection in the live video of table tennis games. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1 1 is a flow chart of a method for real-time detection of multi-task data for table tennis match videos according to an embodiment of the present invention;
[0036] Figure 2 Schematic diagram of the model structure of a multi-task data real-time detection model in an embodiment of the present invention;
[0037] Figure 3 2 is a schematic diagram of a multi-task data real-time detection device for table tennis match videos according to an embodiment of the present invention;
[0038] Figure 4 2 is a schematic diagram of a multi-task data visualization real-time detection result of ball detection and hitting action recognition in an embodiment of the method of the present invention;
[0039] Figure 5 This is a schematic diagram of the visualization results obtained using the TTNet method in Experimental Example 2 of the present invention;
[0040] Figure 6This is a schematic diagram of the visualization results obtained using the TTNet improved method in Experimental Example 2 of the present invention. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.
[0042] The inventive concept of the present invention is: in view of the fact that the existing technology cannot accurately and efficiently complete multiple target tasks of ball detection and hitting action recognition for live videos of table tennis matches, the embodiments of the present invention provide a multi-task data real-time detection method and device for table tennis match videos, which can achieve efficient and accurate visual real-time detection.
[0043] Figure 1 FIG. 1 is a flow chart of a multi-task data real-time detection method for table tennis game videos according to an embodiment of the present invention. Figure 1 As shown, the embodiment provides a multi-task data real-time detection method for table tennis game videos, including the following steps:
[0044] S1, builds a multi-task data real-time detection model, including a video feature extraction module, a ball detection module, and a hitting action recognition module.
[0045] Specifically, if Figure 2 As shown in the figure, the video feature extraction module includes a three-dimensional convolutional layer, a maximum pooling layer and multiple convolutional layers connected in sequence; the ball detection module includes multiple convolutional layers, an upsampling layer and a convolutional layer connected in sequence, and the multiple convolutional layers of the ball detection module are jump-connected with the maximum pooling layer and multiple convolutional layers in the video feature extraction module; the batting action recognition module includes a maximum pooling + downsampling layer, a Transformer encoder and a linear layer connected in sequence.
[0046] Specifically, the multiple convolutional layers of the video feature extraction module are the first convolutional layer, the second convolutional layer and the third convolutional layer, and the multiple convolutional layers in the ball detection module are the fourth convolutional layer, the fifth convolutional layer and the sixth convolutional layer. In this embodiment, the jump connection method is that the maximum pooling layer is connected to the sixth convolutional layer, the first convolutional layer is connected to the fifth convolutional layer, and the second convolutional layer is connected to the fourth convolutional layer.
[0047] S2, perform unsupervised pre-training on the multi-task data real-time detection model to obtain pre-training parameters, and perform supervised training based on the pre-training parameters and using the labeled table tennis game video dataset.
[0048] Specifically, the contrastive learning method is used to perform unsupervised pre-training on the video feature extraction module to learn basic visual perception capabilities, so that the frame-by-frame visual features extracted by the video feature extraction module have multi-task generalization and pre-training parameters are obtained; based on the pre-training parameters, a labeled table tennis match video dataset is used to perform supervised training on the multi-task data real-time detection model, so that the model can further adapt to the task scenarios of live table tennis match videos, and is more sensitive to the fast-moving table tennis ball and the hidden batting actions of the players in table tennis matches. The binary cross-entropy loss function is used to add the loss functions of the two tasks of ball detection and batting action recognition to obtain the loss function of the supervised training stage, and the multi-task data real-time detection model is optimized.
[0049] S3, uses the deep neural network with a three-dimensional convolutional layer added to the trained video feature extraction module to extract the frame-by-frame visual features of the table tennis game video.
[0050] Specifically, for the RGB three-channel input of each frame of the video, we first use a three-dimensional convolution layer with a convolution kernel size of 5×5×5 and a step size of 1 to downsample the temporal and spatial dimensions to obtain a 64-channel feature vector, and then further downsample it through a maximum pooling layer with a convolution kernel size of 2 and a step size of 2. Finally, we use multiple convolution layers of the ResNet-50 residual convolutional neural network to extract the frame-by-frame visual features.
[0051] S4 uses the trained ball detection module to convert the frame-by-frame visual features into a single-channel confidence heat map to determine the frame-by-frame position of the table tennis ball.
[0052] Specifically, the ball detection module receives the intermediate outputs of the maximum pooling layer and multiple convolution layers in the video feature extraction module through multiple convolution layers respectively, and then performs upsampling and convolution operations in the spatial dimension through the upsampling layer and convolution layer. The frame-by-frame visual features output by the video feature extraction module are converted into a heat map of the confidence of the table tennis ball detection at each position in the single-channel picture, and then the frame-by-frame position of the table tennis ball is determined as the ball detection result.
[0053] S5, using the trained batting action recognition module to reduce the dimension of each frame's visual features and perform position encoding, and after further dimensionality reduction, linear regression is used to obtain the player's batting confidence frame by frame to determine the batting time and number of batting times.
[0054] Specifically, the 2048-channel frame-by-frame visual features output by the video feature extraction module ResNet-50 are downsampled through the maximum pooling + downsampling layer to output 256-dimensional low-dimensional features. The low-dimensional features are positionally encoded using the Transformer encoder to further model the temporal and spatial information. The encoded 256-dimensional low-dimensional features are downsampled again through the linear layer to obtain 128-dimensional features. Linear regression is then performed to obtain the player's batting confidence frame by frame. The confidence reflects the probability of a batting event in each frame, thereby determining the batting moment and number of batting times.
[0055] S6, the output results of the ball detection module and the batting action recognition module are visualized through drawing tools and annotated in the table tennis game video, realizing multi-task data visualization and real-time detection of ball detection and batting action recognition.
[0056] Specifically, during visualization, the position of the table tennis ball is marked on the video in the form of a point, and curves of different colors are used to represent the probability of the hitting event of the upper player and the lower player respectively. Figure 4 As shown, a series of points clearly show the trajectory of the table tennis ball. At the same time, we can also see that the player above is about to hit the ball, and the position of this frame on the curve graph is close to the peak of the curve.
[0057] In summary, a multi-task data real-time detection method for table tennis match videos constructs a multi-task data real-time detection model by adding a deep neural network with a three-dimensional convolutional layer, adds two task modules of ball detection and hitting action recognition, abandons time-consuming methods such as extracting optical flow, and the model depth is also controlled. After experimental verification, it can achieve high-precision real-time detection of more than 50FPS in live table tennis match videos. The method of the present invention can improve the ability of sports channels and viewing software to perform real-time data analysis and visualization during live broadcasts, and has certain commercial application prospects. In addition, the multi-task data real-time detection model provided by the method of the present invention is also applicable to the analysis of other racket sports and has a wide range of application value.
[0058] Based on the same inventive concept, the embodiment further provides a multi-task data real-time detection device 300 for table tennis game videos, such as Figure 3 As shown, it includes a model building unit 301, a model training unit 302, a video feature extraction unit 303, a ball detection unit 304, a hitting action recognition unit 305 and a visualization generation unit 306.
[0059] The model building unit 301 is used to build a multi-task data real-time detection model, including a video feature extraction module, a ball detection module and a hitting action recognition module;
[0060] The model training unit 302 is used to perform unsupervised pre-training on the multi-task data real-time detection model to obtain pre-training parameters, and perform supervised training based on the pre-training parameters and using the labeled table tennis game video dataset;
[0061] The video feature extraction unit 303 is used to extract frame-by-frame visual features of the table tennis game video using a deep neural network with a three-dimensional convolutional layer added to the trained video feature extraction module;
[0062] The ball detection unit 304 is used to use the trained ball detection module to convert the frame-by-frame visual features into a single-channel confidence heat map to determine the frame-by-frame position of the table tennis ball;
[0063] The hitting action recognition unit 305 is used to use the trained hitting action recognition module to reduce the dimension of the visual features frame by frame and perform position encoding. After further dimensionality reduction, linear regression is performed to obtain the player's hitting confidence frame by frame, thereby determining the hitting time and number of hits;
[0064] The visualization generation unit 306 is used to visualize the output results of the ball detection module and the hitting action recognition module through drawing tools, and mark them in the table tennis game video to achieve multi-task data visualization and real-time detection of ball detection and hitting action recognition.
[0065] It should be noted that the multi-task data real-time detection device for table tennis match videos provided in the above embodiment and the multi-task data real-time detection method for table tennis match videos belong to the same concept. The specific implementation process is detailed in the multi-task data real-time detection method for table tennis match videos, which will not be repeated here.
[0066] To verify the effectiveness of the method of the present invention, the following ablation experiments and comparative experiments were conducted. The experimental equipment used was a server with a 64-core AMD EPYC 7763 64-Core Processor CPU, an NVIDIA A100 GPU and 500G memory.
[0067] First, we introduce three benchmark methods used in the following experimental examples: (1) TTNet method: The table tennis real-time spatiotemporal video analysis network (TTNet) extracts video features through a two-dimensional convolutional neural network, performs semantic segmentation, and finally realizes two-stage ball detection and hitting event detection; (2) CARL improved method: The frame-level video encoder (FVE) is introduced in the comparative action representation learning framework (CARL) to model the temporal context. The working process of FVE is to first extract a spatial feature of size T×224×224×3 of the RGB video sequence through a two-dimensional grid (ResNet-50), and then use a transformer to project the extracted spatial feature to a middle-level feature of size T×256. The feature will then be encoded and further input into the Transformer encoder for modeling. Finally, a linear layer is used to obtain a frame-by-frame representation of the video. The improved CARL method is to input the features into a series of upsampling layers based on CARL's focus on frame-by-frame video feature extraction, and use the skip-connection structure of UNet to restore the features to the same size as the original input image, thereby obtaining a heat map for ball detection and adding ball detection and batting action recognition functions; (3) I3D+LSTM method: The I3D model is a convolutional neural network containing two streams. It takes video and optical flow as input and outputs the category of the video to achieve feature modeling. In order to support action recognition in table tennis scenes, a structure combining I3D with long-term short-term neural network (LSTM) is designed. The video features output by I3D are input into LSTM to detect the batting action of table tennis. However, I3D severely downsamples the video in the spatial dimension and cannot detect subtle objects such as balls. In addition, I3D needs to extract optical flow and cannot achieve real-time performance.
[0068] Experimental Example 1
[0069] Ablation experiments were conducted to compare the performance of the improved CARL method and the proposed method in ball detection and shot event detection. The improved CARL method and the proposed method were trained for 50 rounds each. The accuracy of ball detection and shot event detection is shown in Table 1. In Table 1, top_1 refers to the accuracy of the result with the highest predicted probability matching the actual result, and top_5 refers to the accuracy of the top five predicted probability matching the actual result. Table 1 also shows the recall and precision of the upper and lower player shot events.
[0070] Table 1 Experimental results of various methods in ball detection and hitting event detection
[0071]
[0072] Experimental results indicate that the proposed method outperforms the improved CARL method in both ball detection and shot event detection. The primary difference between the two methods lies in the replacement of the two-dimensional convolutional feature extraction network with a three-dimensional convolutional feature extraction network, effectively taking the relationship between adjacent frames into account during initial feature extraction. For table tennis match videos, the position of the ball can shift significantly due to the significant information variation between frames. The proposed method, which utilizes a three-dimensional convolutional feature extraction network in its video feature extraction module, extracts frame-by-frame features from table tennis match videos for detailed modeling, significantly improving both ball detection and shot event performance.
[0073] Experimental Example 2
[0074] Comparative experiments were conducted. First, the TTNet method was compared with the method of the present invention in multi-task detection of ball detection and hitting event detection. Second, the I3D+LSTM method was compared with the method of the present invention in hitting event detection. Third, the I3D+LSTM method was compared with the method of the present invention in terms of real-time performance.
[0075] First, the test results of the TTNet model are visualized and compared with the test results of the method of the present invention. TTNet is trained on the same dataset, and samples are randomly selected and visualized. The results are as follows: Figure 5 As shown in the figure, the prediction results of the ball position and the hitting event are not ideal. The predicted results of the event do not correspond to the actual situation, and the position of the ball is not predicted. In order to enable the model to learn the temporal information of the video, the TTNet method was improved. In the experiment, a dual-stream structure was introduced to TTNet, that is, video frames and optical flow were used as inputs of the model. After training on the same dataset, the results are shown below. Figure 6 As shown in Figure 2, the effect of the improved method using TTNet has not been significantly improved. The prediction probability of the hitting event has always remained between 0.6 and 0.7, and the hitting event cannot be predicted correctly. At the same time, the prediction of the ball position is still not accurate enough. Figure 6 As can be seen in the figure, there are many errors. Therefore, neither TTNet nor the improved TTNet can effectively detect ball and hitting events in tennis game videos. In particular, for the hitting event detection task, the model cannot learn effective weights.
[0076] Secondly, the I3D+LSTM method was compared with the method of the present invention in terms of shot detection. The experimental results of the method of the present invention are shown in Table 1. The I3D+LSTM method achieved a recall rate and precision of 0.88 and 0.86 for the upper player's shot event, and a recall rate and precision of 0.86 and 0.90 for the lower player's shot event. This shows that the I3D+LSTM method can effectively extract high-dimensional features of the game and detect shot actions. The method of the present invention outperforms the I3D+LSTM method in all dimensions.
[0077] Third, regarding real-time performance, experimental testing has shown that the I3D+LSTM method, while relatively robust for the single task of identifying ball-hitting events, uses RGB frames and optical flow as input, requiring pre-calculation of the optical flow. Under current conditions, optical flow calculations are time-consuming and similarly cannot meet the speed requirements for real-time processing of live video. However, the method presented in this paper can achieve a processing speed of 53.15 frames per second when processing table tennis match videos, thus enabling real-time processing of live video.
[0078] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A multi-task data real-time detection method for table tennis game videos, characterized in that: The following steps are involved: A multi-task data real-time detection model was constructed, including a video feature extraction module, a ball detection module, and a batting motion recognition module. The video feature extraction module includes a sequentially connected 3D convolutional layer, a maximum pooling layer, and multiple convolutional layers. The ball detection module includes sequentially connected multiple convolutional layers, an upsampling layer, and a convolutional layer. The multiple convolutional layers of the ball detection module are jump-connected to the maximum pooling layer and multiple convolutional layers in the video feature extraction module. The batting motion recognition module includes sequentially connected maximum pooling + downsampling layers, a Transformer encoder, and a linear layer. Unsupervised pre-training of the multi-task data real-time detection model is performed to obtain pre-training parameters. Based on the pre-training parameters, supervised training is performed using a labeled table tennis game video dataset. Utilize a deep neural network with a 3D convolutional layer added to the trained video feature extraction module to extract frame-by-frame visual features of table tennis match videos; The trained ball detection module is used to convert the frame-by-frame visual features into a single-channel confidence heat map to determine the frame-by-frame position of the table tennis ball. The trained batting action recognition module is used to reduce the dimensionality of the frame-by-frame visual features and perform position encoding. After further dimensionality reduction, linear regression is used to obtain the player's batting confidence frame by frame to determine the batting time and number of batting times. The output results of the ball detection module and the batting action recognition module are visualized using drawing tools and annotated in the table tennis game video, realizing real-time visualization of multi-task data for ball detection and batting action recognition.
2. The multi-task data real-time detection method for table tennis game videos according to claim 1 is characterized in that: The unsupervised pre-training of the multi-task data real-time detection model to obtain pre-training parameters includes: unsupervised pre-training of the video feature extraction module using a contrastive learning method to learn basic visual perception capabilities so that the frame-by-frame visual features extracted by the video feature extraction module have multi-task generalization, thereby obtaining pre-training parameters; The supervised training based on pre-training parameters and using a labeled table tennis match video dataset includes: supervised training of a multi-task data real-time detection model using the labeled table tennis match video dataset based on the pre-training parameters, so that the model is further adapted to the task scenario of a live table tennis match video, and is more sensitive to the fast-moving table tennis ball and the hidden hitting movements of the players in the table tennis match; a binary cross-entropy loss function is used to add the loss functions of the two tasks of ball detection and hitting movement recognition to obtain a loss function in a supervised training stage, and the multi-task data real-time detection model is optimized.
3. The multi-task data real-time detection method for table tennis game videos according to claim 1, characterized in that: The method of extracting frame-by-frame visual features of a table tennis match video by using a deep neural network with a three-dimensional convolutional layer added to a trained video feature extraction module includes: For the RGB three-channel input of each frame of the video, a three-dimensional convolution layer is first used to downsample the time and space dimensions to obtain a feature vector, which is then further downsampled through a maximum pooling layer. Finally, multiple convolution layers are used to extract the frame-by-frame visual features.
4. The multi-task data real-time detection method for table tennis game videos according to claim 1, characterized in that: The method of using the trained ball detection module to convert frame-by-frame visual features into a single-channel confidence heat map to determine the frame-by-frame position of the table tennis ball includes: The ball detection module receives the intermediate outputs of the maximum pooling layer and multiple convolutional layers in the video feature extraction module through multiple convolutional layers, and then performs upsampling and convolution operations in the spatial dimension through the upsampling layer and convolution layer. The frame-by-frame visual features output by the video feature extraction module are converted into a heat map of the confidence of the table tennis ball detection at each position in the single-channel picture, and then the frame-by-frame position of the table tennis ball is determined as the ball detection result.
5. The multi-task data real-time detection method for table tennis game videos according to claim 1, characterized in that: The trained batting action recognition module is used to reduce the dimensionality of the frame-by-frame visual features and then perform position encoding. After further dimensionality reduction, linear regression is performed to obtain the player's batting confidence frame by frame, thereby determining the batting time and number of batting times, including: The frame-by-frame visual features output by the video feature extraction module are downsampled through the maximum pooling + downsampling layer to output low-dimensional features. The low-dimensional features are positionally encoded using the Transformer encoder. The encoded low-dimensional features are downsampled through the linear layer and the dimension is further reduced again, and linear regression is performed to obtain the player's batting confidence frame by frame. The confidence reflects the probability of a batting event in each frame, thereby determining the batting moment and number of batting times.
6. A device for real-time detection of multi-task data for table tennis match videos, implemented using the method for real-time detection of multi-task data for table tennis match videos according to any one of claims 1 to 5, characterized in that: It includes a model building unit, a model training unit, a video feature extraction unit, a ball detection unit, a hitting action recognition unit and a visualization generation unit; The model building unit is used to build a multi-task data real-time detection model, including a video feature extraction module, a ball detection module and a hitting action recognition module; The model training unit is used to perform unsupervised pre-training on the multi-task data real-time detection model to obtain pre-training parameters, and perform supervised training based on the pre-training parameters and using the labeled table tennis game video dataset; The video feature extraction unit is used to extract frame-by-frame visual features of the table tennis game video using a deep neural network with a three-dimensional convolutional layer added to the trained video feature extraction module; The ball detection unit is used to convert the frame-by-frame visual features into a single-channel confidence heat map using the trained ball detection module to determine the frame-by-frame position of the table tennis ball; The batting action recognition unit is used to use the trained batting action recognition module to reduce the dimension of the visual features frame by frame and then perform position encoding. After further dimensionality reduction, linear regression is performed to obtain the player's batting confidence frame by frame, thereby determining the batting time and number of batting times. The visualization generation unit is used to visualize the output results of the ball detection module and the hitting action recognition module through a drawing tool, and mark them in the table tennis game video, thereby realizing multi-task data visualization and real-time detection of ball detection and hitting action recognition.