A table tennis movement detection system based on pure visual perception

Abstract

The application discloses a table tennis movement detection system based on pure visual perception, which comprises a table tennis detection and event monitoring module, a human-racket posture estimation module and a time sequence action recognition and quality evaluation module; the table tennis detection and event monitoring module is based on an improved DTTNet model, realizes table tennis trajectory detection and event identification such as touch table, touch net and net passing through deformable convolution of multi-frame video stacking; the human-racket posture estimation module adopts a joint inference structure of a lightweight detection network and a high-resolution posture estimation network, outputs human key points and racket three key point coordinates, and constructs a "human-racket" joint kinematics model; the time sequence action recognition and quality evaluation module fuses multi-view video and posture features, realizes classification and quality evaluation of table tennis technical actions through time sequence boundary matching and multi-scale feature analysis, and has the characteristics of high detection precision and strong real-time performance, and can realize automatic identification and quantitative evaluation of table tennis movement.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a table tennis motion detection system based on pure visual perception. Background Technology

[0002] The training and sports analysis in table tennis have long relied on coaches' manual observation and experience-based judgment. This makes it difficult to accurately capture technical details during high-speed competition, leading to significant biases in movement assessment and problems such as strong subjectivity, inconsistent quantitative standards, and delayed real-time feedback. This experience-driven model has hindered the improvement of training efficiency and the scientific development of table tennis.

[0003] In the field of computer vision technology, some existing visual detection schemes have been applied to sports training, such as 2D pose estimation or temporal action classification models. However, these schemes mostly rely on single-viewpoint or low-frame-rate video input, and have the following technical shortcomings:

[0004] 1. Insufficient accuracy in detecting high-speed small targets: Ping-pong balls are small in size and move at high speed, making them prone to motion blur and occlusion problems, resulting in large trajectory tracking errors;

[0005] 2. Limited ability to model the temporal sequence of complex movements: Existing methods are mostly focused on human joint detection and lack modeling of the "human-racket" joint kinematic model, making it difficult to quantify key parameters of the hitting action (such as swing angle).

[0006] 3. Data gaps: Public datasets lack high-quality, fine-grained annotation data for table tennis movements, especially time-series movement labels that incorporate quality assessments (such as Perfect / Good / OK ratings).

[0007] In addition, existing solutions rely heavily on hardware devices for their business processes. For example, deploying multiple cameras makes it difficult to achieve low-cost, non-intrusive analysis. Other solutions based on wearable sensors interfere with users' natural movements. Summary of the Invention

[0008] The purpose of this invention is to provide a table tennis motion detection system based on pure visual perception to solve the problems mentioned in the background art.

[0009] To address the aforementioned technical problems, this invention provides the following technical solution: a table tennis motion detection system based on pure visual perception, comprising a table tennis detection and event monitoring module, a human-racket posture estimation module, and a temporal action recognition and quality assessment module; the table tennis detection and event monitoring module receives multi-view video streams and outputs the motion trajectory of the table tennis ball and the sequence of hitting events; the human-racket posture estimation module receives the multi-view video streams and outputs the coordinates of the user's human body key points and the coordinates of the racket key points; the temporal action recognition and quality assessment module fuses the hitting event sequence, the human body key point coordinates, and the racket key point coordinates to output the user's technical action classification and quality score.

[0010] Preferably, the improved DTTNet model based on deep learning includes: through multi-frame spatiotemporal joint modeling and a unified two-stage processing mechanism, the generated features are highly shared and reused in parallel to achieve the localization and tracking of ping-pong balls and the monitoring of key events.

[0011] Preferably, the table tennis key event monitoring utilizes an event recognition model trained on the publicly available OpenTTGames dataset. During training, a weighted cross-entropy loss function is used to address the class imbalance problem, and an event smoothing labeling strategy is introduced. When constructing the labels for the training dataset, for the manually labeled event center frame, the probability weight of the event is propagated to the four frames before and after it, so that the true label weights of adjacent frames are smoothly decayed, rather than being set as hard labels. The value of the decay weight is calculated according to the form of a sine function.

[0012] Preferably, the implementation of the ping-pong ball detection and event monitoring module includes:

[0013] Step S11: Perform feature processing using the spatiotemporal context information of the target frame and its preceding and following frames to construct the input tensor;

[0014] Step S12: Extract low-level features based on the input tensor, generate globally shared features, and obtain preliminary coordinates of the coarse localization result;

[0015] Step S13: Based on the preliminary coordinates of the coarse localization results, the original image is locally cropped. The cropped image is then standardized and used as the input to the network for refined detection, outputting accurate shared features.

[0016] Step S14: Use the globally shared features and the precisely shared features as input features for table tennis detection and table tennis key event monitoring, and output fine coordinates;

[0017] Step S15: Calculate the ping-pong ball detection position in the original image by mapping the preliminary coordinates and the fine coordinates according to their positional relationship;

[0018] Step S16: By reusing the refined detection output of precise shared features, extract local refined features, and splice and fuse them with global event features as the final input of the classification head, the video frame is classified into four categories of table tennis events, including table tennis bounce, net touch, net over, and empty events.

[0019] Preferably, the human-racket joint pose estimation module constructs a joint inference architecture comprising a PicoDet object detection model and an HRNet keypoint detection model. The PicoDet object detection model includes: an ESNet backbone network, used to extract multi-scale features stepwise through depthwise separable convolutions and channel shuffling; the HRNet model adopts a multi-resolution parallel branch architecture, repeatedly exchanging and fusing features on different resolution sub-networks, outputting features from the highest resolution branch and generating a keypoint heatmap; a lightweight object detection network is used to locate the user's location; and a high-resolution convolutional neural network is used to regress the keypoint coordinates of the human body and racket.

[0020] Preferably, the human-racket posture estimation module further includes: training using a dedicated dataset, which is supplemented with annotation information for three key points of the racket: the starting point of the handle, the end of the grip, and the center of the racket face, to uniquely represent the spatial posture of the racket; and defining skeletal connections for table tennis motion analysis, including: head posture connection: connecting the key points of the eyes and ears to form the basis of head posture estimation; trunk and limb connection: connecting the key points of the shoulder, elbow, wrist, hip, knee, and ankle to form the kinematic chain of the main limbs; human-racket association connection: connecting the key points of the wrist and the starting point of the handle to establish the interaction between the user and the equipment; and racket vector axis connection: sequentially connecting the key points of the starting point of the handle, the end of the grip, and the center of the racket face to construct a racket reference system for calculating the racket face orientation and swing trajectory.

[0021] Preferably, the implementation of the human-racket pose estimation module includes:

[0022] Step S21: Extract initial feature maps using the input video frames, and then extract features at different scales based on the initial feature maps;

[0023] Step S22: Use CSP-PAN feature pyramid to fuse features extracted at different scales and output a multi-level feature map with a unified channel dimension;

[0024] Step S23: Predict the user's bounding box based on multi-level feature maps;

[0025] Step S24: Use the predicted bounding box to crop the target region from the original input to form a standardized input block;

[0026] Step S25: Perform multi-resolution feature fusion based on standardized input blocks to generate key points;

[0027] Step S26: Decode the generated key points to generate standardized structured pose information.

[0028] Preferably, the temporal action recognition and quality assessment module uses a boundary matching network to receive multi-view video streams and outputs the start and end frames of the technical actions; it extracts and outputs the user's core joint coordinates and kinematic parameters through key body features; and it uses a SlowFast dual-path architecture to fuse key body features and video spatiotemporal features to output the classification results and action quality assessment results of table tennis technical actions, covering the three major technical systems of forehand, backhand, and chop.

[0029] Preferably, the implementation of the timing action recognition and quality assessment module includes:

[0030] Step S31: Using the user region image patch after being cropped by the bounding box output by the human-racket pose estimation module in the multi-view video stream as input, a pre-trained dual-stream network is used to extract the spatiotemporal joint feature sequence;

[0031] Step S32: Through convolution and the Sigmoid activation function, predict the probability sequence of each time point as the start / end of the action in real time, generate initial action proposals using a sliding window, and screen the quality of the initial proposals through boundary matching sampling and double confidence calculation;

[0032] Step S33: Establish a personalized reference coordinate system and extract key body features; using the posture estimation data output by the lightweight HRNet model as input, extract the x / y coordinates of the core joints of the user's wrist, elbow, and shoulders, generate an 8-dimensional spatiotemporal feature vector, introduce a dynamic time warping algorithm, eliminate the differences in movement rhythm among different users through nonlinear time alignment, construct a dynamic triangle analysis unit with shoulder-elbow-wrist as the core, calculate the triangle area change rate, characterize the upper limb linkage mode, additionally extract joint angular velocity difference features and normalized motion velocity, enhance the temporal sensitivity and individual adaptability of features, and provide kinematic feature input;

[0033] Step S34: Identify and assess the quality of the actions, and output high-scoring action sequence segments.

[0034] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0035] 1. The detection accuracy of this invention is improved: By combining the improved DTTNet model with DCN deformable convolution technology, high-speed trajectory tracking of ping-pong balls is achieved. Employing a nine-frame continuous video input and a two-stage detection strategy effectively solves the problems of ball motion blur and background interference.

[0036] 2. The motion analysis dimensions of this invention are more comprehensive: it defines three key points of the racket, constructs a "human-racket" joint kinematic model, and adopts a SlowFast dual-path network to achieve three-level evaluation of motion quality.

[0037] 3. The present invention is more cost-effective: it adopts a pure vision non-contact analysis scheme, avoiding the high cost of traditional wearable sensors or fixed hardware devices. Attached Figure Description

[0038] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0039] Figure 1 This is an overall architecture diagram of the table tennis motion detection system provided in an embodiment of the present invention.

[0040] Figure 2 This is a core architecture diagram of the ping-pong ball detection and event monitoring module provided in an embodiment of the present invention.

[0041] Figure 3 This is a core architecture diagram of the human-racket posture estimation module provided in an embodiment of the present invention.

[0042] Figure 4 This is a core architecture diagram of the timing action recognition and quality assessment module provided in an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Combination Figure 1 As shown, the present invention provides a technical solution: a table tennis motion detection system based on pure visual perception, comprising: a table tennis detection and event monitoring module, a human-racket posture estimation module, and a temporal motion recognition and quality assessment module;

[0045] In this embodiment, the present invention is based on the DTTNet architecture to build a ping-pong ball detection and event monitoring module. The core technology of this module lies in achieving high feature sharing and parallel reuse through multi-frame spatiotemporal joint modeling and a unified two-stage (global → precise) processing mechanism, thereby efficiently and accurately processing the positioning and tracking of ping-pong balls and the monitoring of key events.

[0046] For example, constructing a ping-pong ball detection and event monitoring module includes:

[0047] The ping-pong ball detection and event monitoring module first constructs the spatiotemporal input and initializes the features: For a certain frame of video, the system receives a sequence of 9 consecutive original video frames (the frame and the 4 frames before and after it). Each frame is first reshaped and downsampled to make its shape uniform to 320×128×3, and then spliced ​​along the channel dimension to construct an input tensor of 320×128×27.

[0048] The table tennis detection and event monitoring module enters the global detection phase: the tensor extracts low-level features from the VGG type encoder to generate globally shared features. Globally shared features Based on the labeled ping-pong ball coordinates, a Gaussian distribution of the ping-pong ball's horizontal and vertical coordinate positions is obtained through multilayer perceptron regression. The maximum value of the Gaussian distribution is then taken to obtain the preliminary coordinates. and Globally shared features It contains rich spatiotemporal information, which can effectively solve the motion fuzziness problem in table tennis detection, and can be used as one of the input features for the two downstream tasks of table tennis detection and table tennis event monitoring.

[0049] Preliminary coordinates based on coarse positioning results and The ping-pong ball detection and event monitoring module enters the precise detection stage: First, the coarse coordinates are scaled back to the original resolution to determine the center point of the cropped region. Using this as a reference, the original image is locally cropped and then reshaped to 320×128×3 dimensions. The standardized cropped image is then used as input to the network for refined detection. This precise detection network also uses a VGG-type encoder, but its two upper ordinary CNN convolutional layers are replaced with deformable convolutional network (DCN) modules. These modules dynamically adjust the sampling positions to adapt to the small size and complex trajectory of the ping-pong ball target. This stage ultimately completes the regression of precise coordinates and outputs precise shared features. .

[0050] Globally shared features and precise feature sharing The table tennis features serve as input for both downstream tasks: table tennis detection and event monitoring. The table tennis detection component utilizes precisely shared features. Then, a linear regression layer is added to obtain the Gaussian distribution of the horizontal and vertical coordinates of the ping-pong ball. The maximum value of the Gaussian distribution is then taken to obtain the fine coordinates. and , refine coordinates and with preliminary coordinates and Based on the positional relationship of the mapping, the detection position of the ping-pong ball in the original image is calculated. and During this process, the table tennis event monitoring task also enters the final classification stage: it reuses features from the precise detection stage, extracts local refined features, and concatenates and fuses them with global event features as the final input to the classification head. To ensure robustness, a weighted cross-entropy loss function is used to address the class imbalance problem, and a sinusoidal smoothing annotation strategy is applied to enhance the ability to identify event temporal boundaries. Ultimately, each frame of video is classified into four types of table tennis events: touching the table and bouncing, touching the net, crossing the net, and empty space events.

[0051] In this embodiment, as Figure 2 As shown, the working process of the ping-pong ball detection and event monitoring module is as follows:

[0052] Step S11: Construct multi-frame spatiotemporal input.

[0053] (1) Target video frame Four frames before and after it are selected, for a total of nine frames, to form a time stack:

[0054] {

[0055] in, Indicates the last 4 frames. This indicates the first 4 frames.

[0056] (2) Each frame of the image is downsampled and reshaped to a uniform size of 320×128×3, and then stitched together along the channel dimension as follows:

[0057]

[0058] in Represents a tensor. The tensor has a shape of 320×128×3.

[0059] (3) Normalize and standardize the input tensor to form the network input.

[0060] Step S12: Perform global feature extraction.

[0061] In this embodiment, the tensor Input to a VGG-style encoder to extract global features. :

[0062]

[0063] For each coordinate dimension, define a one-dimensional Gaussian target distribution:

[0064]

[0065]

[0066] in( The coordinates of the labeled points are the true coordinates, which are converted into Gaussian distribution probabilities at a uniform size (320×128). Based on experience with the pixel size of a ping-pong ball, a value of 3 is chosen. , Let represent the probabilities of a one-dimensional Gaussian target distribution. These are the Gaussian distribution probabilities after the point coordinates are converted to a uniform size (320×128), where It is a Gaussian distribution probability along the horizontal axis (x-axis). It is a Gaussian distribution along the vertical axis (y-axis); , These represent the coordinates on the x-axis and y-axis, respectively.

[0067] Global features Through a linear layer and a classification head, the final output consists of two tensors. and , The dimension is 320. The dimension is 128.

[0068] ,

[0069] The predicted tensor and the true coordinates are then optimized using distributed cross-entropy loss, and the modulus is changed by reverse propagation:

[0070]

[0071] in, Represents the magnitude of the horizontal coordinate (x-axis), which is used as a normalization term in the loss function. Appear, Represents the position on a uniform horizontal axis. The model predicts a one-dimensional Gaussian distribution probability with a dimension of 320. The y-axis represents the magnitude of the vertical axis and is used as a normalization term in the loss function. Appear, Indicates the position on a vertical axis of uniform size. The model predicts a one-dimensional Gaussian distribution probability with a dimension of 128.

[0072] Take the location of the maximum value of the distribution as the coordinates of the coarse positioning result. :

[0073]

[0074] Step S13: Extract fine features.

[0075] In this embodiment, the coordinates of the coarse localization result obtained in the global detection stage are... Mapping back to the original image resolution The following is specifically expressed as:

[0076]

[0077] Coordinates of coarse positioning results Centered on the original image Crop local area The size of the region is determined by empirical parameters ( Controls are implemented to cover the possible range of motion of the table tennis ball.

[0078]

[0079] The cropped result is reshaped to a uniform input size of 320×128×3, and after normalization preprocessing, it is input into the accurate detection network.

[0080] The cropped image is also fed into a VGG-style encoder for feature extraction. In the first two convolutional layers, deformable convolutional modules are used to replace the fixed sampling mechanism of traditional convolutional layers, so that the convolutional kernel can dynamically adjust the sampling position according to the input features.

[0081] Let the output of a regular convolution be represented as:

[0082]

[0083] in, For the corresponding output position, Sampling location, The sampling grid for the standard convolution kernel (e.g., 3×3). ={(-1,-1),...,(1,1)}), For convolution weights, For the input feature map, in DCN, for each sampling location Learning a two-dimensional offset Defined as:

[0084]

[0085] in Predicted by additional convolutional layers;

[0086]

[0087] This allows the receptive field center of the convolutional kernel to be no longer fixed at each pixel, but rather to adaptively adjust according to the target's motion direction and speed. This enables it to maintain high-response feature focusing even when the ping-pong ball is moving at high speed, experiencing ghosting, or deforming. The final result is the feature extracted in the fine-tuning stage. .

[0088] Step S14: Inspect the ping-pong ball.

[0089] In this embodiment, the features obtained in the fine-tuning stage are... Through a linear layer and a classification head, the final output consists of two tensors. and , The dimension is 320. The dimension is 128.

[0090] The predicted tensor and the true coordinates are then optimized using distributed cross-entropy loss, and the modulus is changed by reverse propagation.

[0091] The location of the predicted maximum value is taken as the coordinate of the fine-stage localization result. , means as follows:

[0092]

[0093] The final output position is calculated based on the mapping relationship between global and local results:

[0094]

[0095] in, The center of the clipping is the coarse coordinate obtained in the global stage. Original frame size This is the downsampling size. , To crop the window width and height, These are the local predicted coordinates, i.e., the coordinates obtained in the fine-tuning stage.

[0096] Obtain the final coordinates of the sphere's center. .

[0097] Step S15: Monitor the table tennis incident.

[0098] In this embodiment, globally shared features are used. with local fine features By splicing and merging along the channel dimension, the fusion feature is obtained. :

[0099]

[0100] Fusion features Perform global average pooling (GAP) to obtain the frame-level representation vector. , as the input for the classification header.

[0101] Predicting event category probabilities using fully connected layers and the Softmax function. :

[0102]

[0103]

[0104]

[0105] Among them, event categories There are four categories: bounces after hitting the platform. Internet , over the net and empty events , This represents the weight matrix used to predict the probability of event categories in a fully connected layer for event classification. This represents the bias term used to predict the probability of an event category in a fully connected layer for event classification.

[0106] For example, a sinusoidal smoothing annotation mechanism is introduced to enhance temporal continuity: To address issues such as blurred event boundaries and uncertain hitting times in manual annotation, this invention uses a sine function to generate temporally smoothed labels, specifically for events occurring in the center frame. The event, whose tag value in a neighboring frame is defined as:

[0107]

[0108] in, For soft labels generated based on a sinusoidal smoothing strategy, representing the first... An event in the frame The probability weights on; The frame number of the video frame currently being tagged. Indicates the first manually labeled The frame number of the center frame where the event occurred. The smoothing window length is the window size used to define the smoothing range. In this invention, it is set to 8, which represents the event center frame. front and back The labels of the frames (i.e., the four frames before and after) are smoothly decayed. This method creates a continuous weight decay distribution of labels before and after the event, providing a progressively increasing supervision signal over time and avoiding training instability caused by single-frame label jumps. This allows the network to perceive the dynamic process of the start and end of the event more smoothly over time.

[0109] Using weighted cross-entropy loss to address class imbalance: Since different event types occur with significantly different frequencies, this invention introduces class weights during training to balance the gradient contributions of different sample types. The loss function is defined as follows: :

[0110]

[0111] Where N represents the total number of frames involved in the calculation. The probability of the event predicted by the model; For category weights, we assign them as follows:

[0112]

[0113] Finally, the ping-pong event category corresponding to a certain frame is obtained. .

[0114]

[0115] in, Indicates the probability of the event category that occurred.

[0116] Ultimately, the system uses frame sequence numbers. Event Category Constructing the output event sequence :

[0117]

[0118] In this embodiment, the present invention constructs an efficient and accurate human-racket joint pose estimation module based on the PicoDet object detection model and the HRNet keypoint detection model. The human-racket pose estimation module combines the robustness of object detection with the high accuracy of pose estimation through an innovative joint inference framework. First, it locates the user's position in real time, and then regresses the precise coordinates of 17 standard human keypoints and 3 custom racket keypoints within the detection area, laying a reliable data foundation for subsequent motion quality assessment.

[0119] Specifically, the human-racket pose estimation module first processes the input video frames using the PicoDet model. This process employs a lightweight design, with its backbone network ESNet being a lightweight and efficient streaming feature extraction network. Through a 13-stage ESBlock structure (Stage 1: x3, Stage 2: x7, Stage 3: x3), it extracts multi-scale features (C3, C4, C5) level by level using depthwise separable convolutions and channel shuffling. Subsequently, a lightweight CSP-PAN feature pyramid fuses these multi-scale features into a unified 96-channel pyramid feature structure (P3-P6) through a bidirectional (Top-Down and Bottom-Up) feature fusion path. Finally, the coupled detection head regresses the user's precise bounding box coordinates based on the fused features, providing a spatial localization foundation for subsequent keypoint detection.

[0120] By cropping the image region corresponding to the obtained bounding box coordinates from the original image and performing normalized scaling, a uniform-sized input image patch is formed. This step aims to eliminate scale differences caused by varying user distances from the camera and focus computational power on the target region, providing scale-normalized input to the HRNet model and ensuring the accuracy and stability of keypoint detection.

[0121] The preprocessed image patches are input into the HRNet model, which regresses to generate a heatmap of 20 key points. HRNet employs a unique parallel multi-resolution branch architecture, with its backbone network consisting of four progressively expanding stages (Stage 1: 1 branch, Stage 2: 2 branches, Stage 3: 3 branches, Stage 4: 4 branches). By repeatedly performing cross-resolution feature exchange and fusion on subnetworks at different resolutions (1 / 4, 1 / 8, 1 / 16, 1 / 32) (upsampling uses bilinear interpolation + 1×1 convolution, downsampling uses strided convolution), the network can always maintain high-resolution representation, thereby preserving spatial detail information to the maximum extent. The final network outputs features from the highest resolution (1 / 4 scale) branch, generates a keypoint heatmap through 1×1 convolution, and decodes the peak positions of the heatmap using ArgMax operations to obtain the precise pixel coordinates of 20 keypoints. These 20 keypoints include 17 human skeleton keypoints from the COCO standard and 3 custom racket keypoints (grip start point, grip end, and racket face center). This design allows the model to not only capture human posture but also accurately track the spatial position and posture of the racket. The output coordinate data of the 20 keypoints (in N×20×2 format, where N is the number of detected users and 2 is the pixel coordinates x, y) forms standardized, structured posture data. This data stream provides a directly callable data interface for subsequent motion quality assessment, technical statistical analysis, and visualization feedback, completing the full conversion from raw video frames to structured motion data.

[0122] In this embodiment, combined with Figure 3 As shown, the working process of the human-racket pose estimation module is as follows:

[0123] Step S21: Perform feature extraction.

[0124] In this embodiment, the input video frames are used. Perform initial feature map Extraction:

[0125]

[0126] Among them, the input video frames After extracting preliminary spatial features through 3×3 convolution, channel compression is performed through 1×1 convolution to obtain the initial feature map. This is used for subsequent multi-scale feature extraction.

[0127] Based on the initial feature map Extracting multi-scale features:

[0128]

[0129] in, , , Corresponding to the lightweight feature extraction block (ESBlock), each Internally, it includes depthwise separable convolution and channel shuffling operations, corresponding to feature maps at scales of 1 / 8, 1 / 16, and 1 / 32. Features extracted at different scales are used for subsequent feature fusion.

[0130] Step S22: Perform feature fusion on the features extracted at different scales.

[0131] In this embodiment, the CSP-PAN feature pyramid is used to fuse features extracted at different scales, unifying the multi-scale features into a 96-channel pyramid feature. :

[0132]

[0133] The function Fuse(·) represents the bidirectional feature fusion operation of the CSP-PAN feature pyramid, which is performed from top to bottom and from bottom to top. By alternating upsampling and downsampling paths, it includes a lightweight CSP module and a bidirectional fusion path, unifying multi-scale features into a 96-channel pyramid feature. This achieves the complementarity of semantic and spatial information.

[0134] For example, the specific formula for the downsampling path is as follows:

[0135]

[0136]

[0137]

[0138] The specific formula for the upsampling path (lower layer → higher layer) is as follows:

[0139]

[0140]

[0141]

[0142] in, , These represent the intermediate fused feature maps generated in the bottom-up path, corresponding to the outputs of the fused low-level and high-level features, respectively; structurally, they inherit from the outputs of the top-down path. Furthermore, it integrates multi-scale contextual information to generate the final pyramid features. Provide input.

[0143] For example, the calculation formula for the CSP module is as follows:

[0144]

[0145] The first 32 channels of the 96-channel input X are processed by a 5×5 depthwise convolution, while the last 64 channels are directly retained. The concatenated channels are then mixed by a 1×1 convolution to obtain the final result.

[0146] Step S23: For the bounding box Make predictions.

[0147]

[0148] In the formula, Head(·) represents the detection head network, which outputs a set of candidate boxes through convolution and classification / regression branches. Each candidate box This corresponds to a detected target area.

[0149] Step S24: Crop and normalize the image region to form a standardized input block. .

[0150]

[0151] Crop(·) indicates that the bounding box is used as a reference. From the original input The target area for cropping is specified; Resize(·,s) represents scaling the cropped area to a fixed size. To eliminate scale differences and form standardized input blocks .

[0152] Step S25: Perform multi-resolution feature fusion and key point regression.

[0153] In this embodiment, multi-resolution feature fusion is performed:

[0154]

[0155] Here, MRF(·) represents a multi-resolution feature fusion structure, containing four resolution branches (1 / 4, 1 / 8, 1 / 16, 1 / 32); it achieves high- and low-level information fusion through cross-resolution feature exchange (upsampling + downsampling), outputting a multi-scale feature set. .

[0156] For example, generating a heatmap of key points:

[0157]

[0158] From the highest resolution features A 20-channel heatmap is generated after 1×1 convolution. There are 20 key points (17 human body key points + 3 racket key points).

[0159] Step S26: Decode the key points and output the structured pose.

[0160]

[0161] By analyzing the heat map Each channel undergoes an ArgMax operation to decode the peak coordinates of the key points, resulting in... ∈ ×20×2; where n is the number of detected targets, 20 is the number of key points, and 2 represents the (x,y) pixel coordinates of each key point, forming standardized structured pose information to provide input for subsequent motion analysis.

[0162] In this embodiment, the implementation of the temporal action recognition and quality assessment module of the present invention is to construct an efficient and accurate temporal action recognition and quality assessment method based on the boundary matching network (BMN) and the SlowFast behavior recognition model. Its core technology lies in achieving a high degree of integration and optimization of action temporal localization, key feature extraction and quality assessment through multi-stage collaborative processing and differentiated feature extraction mechanism.

[0163] In this embodiment, a boundary matching network (BMN) is first used for temporal action localization to accurately determine the start and end frames of the action. Specifically, a pre-trained dual-stream network is used to extract spatiotemporal features from video frames, generating a feature sequence H. Then, two layers of 1D convolution in the basic module are used to further extract shared features, enhancing feature representation capabilities. A unique dual-branch parallel architecture is employed. A temporal evaluation module (TEM) is constructed to predict the start and end point probability sequences of the action in real time, generating initial proposals through a sliding window. A proposal evaluation module (PEM) is then constructed, calculating the confidence score of each potential proposal through a boundary matching mechanism to obtain candidate proposals. Classification and regression confidence scores are then output through 3D and 2D convolutions. Finally, by combining the start / end probabilities and confidence scores, high-scoring proposals are output using Soft-NMS, while low-scoring proposals are filtered out.

[0164] For example, based on temporal localization, the process collects joint coordinates (both shoulders x / y, elbow x / y, wrist x / y) at 10ms intervals using pose estimation data output by a lightweight HRNet model, generates an 8-dimensional spatiotemporal feature vector, and introduces a dynamic time warping (DTW) algorithm for time alignment to construct a dynamic triangular analysis unit with the shoulder-elbow-wrist as the core.

[0165] The action recognition and quality assessment section achieves high-precision action classification based on the output high-scoring action time sequence fragments and extracted key body features through differential processing of spatiotemporal features. The process then normalizes the feature sequence... The corresponding video frame is input into the SlowFast model, and the Slow and Fast paths are processed in parallel:

[0166] The Slow path is mainly used for spatial semantic extraction. The time step is 1 frame sampled every 16 frames, with an input frame rate of 8fps. It adopts the 3DResNet-50 architecture with 2048 channels and extracts spatial features (such as the forward lean posture of the torso in a forehand attack and the wrist pronation shape in a backhand topspin) through 4 layers of residual blocks.

[0167] Fast paths are primarily used for fast motion capture, with a time step size of [missing information]. The entire process does not perform temporal downsampling, employs a lightweight 3D convolutional structure with 256 channels (1 / 8 of the Slow path), and captures fast motion details (such as swing acceleration and racket face angle changes at the moment of impact) through 3 layers of residual blocks. The two paths are laterally connected to achieve feature fusion with a gated recurrent unit (GRU), and the final output is a classification result of table tennis technique movements and a three-level action quality assessment (Perfect / Good / OK).

[0168] In this embodiment, combined with Figure 4 As shown, the working process of the timing action recognition and quality assessment module is as follows:

[0169] Step S31: Perform spatiotemporal feature extraction.

[0170] In this embodiment, the user region image patch after being cropped by the bounding box output by the pose estimation module in the multi-view video stream is used as input, and a pre-trained dual-stream network (spatial stream + temporal stream) is used to extract the spatiotemporal joint feature sequence H:

[0171] Spatial Flow: Perform 3×3 convolution and BatchNorm operations on RGB frames to extract static appearance features such as user body contours and racket shape;

[0172] Time flow: Calculate the optical flow field for adjacent frames and capture dynamic features such as swing trajectory and joint movement trends through 5×5 convolution;

[0173] After concatenating the two-stream features, the feature representation is enhanced by two layers of 1D convolution (kernel size = 3, stride = 1, activation function = ReLU), generating a shared feature sequence with uniform dimensionality. :

[0174]

[0175] Enhanced features Provides input for subsequent two branches.

[0176] Step S32: Perform timing action localization.

[0177] In this embodiment, convolution and the sigmoid activation function are used to predict the probability of each time point being the start / end of an action. :

[0178]

[0179] in, It is the Sigmoid activation function. The learnable convolutional kernel weights are predicted for the start / end point, and the variable k represents the offset of one frame before and after the current time point t (with values ​​of -1, 0, and 1). These are used to capture local temporal features of action boundaries through weighted summation (convolution operation). Indicates a point in time The feature vector (or feature representation) at the location; The bias parameter (learnable parameter) represents the predicted start point (s) or end point (e), used to adjust the offset of the convolution operation result; the output of the convolution operation (linear combination result) is mapped to the [0,1] interval through the Sigmoid activation function, thus obtaining the time point. As a probability value for the start or end of an action.

[0180] Output the initial probability sequence at each time point and the termination probability sequence :

[0181]

[0182]

[0183] A sliding window (window size = 9 frames, step size = 1 frame) is used for traversal. When there are 3 consecutive frames with a probability value > 0.5 within the window, it is marked as the initial action proposal, where is the start frame and is the end frame.

[0184] The screening probability exceeds the threshold Boundary combinations:

[0185]

[0186] in, Indicates boundary combination, Indicates a time interval. Indicates a point in time The probability of being the starting point of an action. express The probability of being the end point of an action.

[0187] For each candidate proposal In the time interval [ , Uniform sampling of N=32 time points within the area:

[0188]

[0189] in, Indicates candidate proposals The corresponding sampled feature sequence, , Represents the original temporal feature sequence. This indicates the number of time points sampled (fixed to 32 here).

[0190] Classification confidence scores are calculated using features through 3D convolution processing. and regression confidence :

[0191]

[0192]

[0193] in, The learnable weight parameters represent the classification task. The learnable bias parameters represent the classification task. This represents the learnable weight parameters for the regression task. This represents the learnable bias parameters for the regression task.

[0194] Calculate the final overall confidence level :

[0195]

[0196] Redundant proposals are suppressed using Soft-NMS, while high-scoring results are preserved.

[0197] Step S33: Extract key body features.

[0198] In this embodiment, joint coordinates (both shoulders) are acquired at 10ms intervals. ,elbow ,wrist An 8-dimensional spatiotemporal feature vector was constructed. The Dynamic Time Warping (DTW) algorithm was used to effectively solve the problem of differences in user movement rhythm through nonlinear time alignment, establish a personalized reference coordinate system, and extract biomechanical features.

[0199] Construct an analysis unit of a dynamic triangle formed by the shoulder-elbow-wrist joints and calculate the area of ​​the triangle. :

[0200]

[0201] in, These represent the x-axis and y-axis coordinates of the shoulder joint in the image or motion capture coordinate system, respectively. and Let x and y represent the x-coordinates and y-coordinates of the elbow joint in the same coordinate system, respectively, and let y be a vertex of the triangle. and These represent the x-axis and y-axis coordinates of the wrist joint in the same coordinate system, respectively.

[0202] Using the Perfect-level actions from the dataset as standard templates, the minimum distance between the input feature sequence and the template is calculated using DTW (Dynamic Time Warping), the temporal sequence is dynamically aligned, and the distance matrix is ​​calculated. :

[0203]

[0204] in, For input features, Template features;

[0205] Calculate cumulative distance :

[0206]

[0207]

[0208] in, This represents the cumulative distance, i.e., the distance between the first few iterations of the input feature sequence. The first feature and template feature sequence The minimum total distance generated when aligning features.

[0209] Path backtracking: Obtain the optimal alignment path and output the normalized feature sequence. .

[0210] For extreme individual differences (such as left-handed players), a hybrid compensation scheme is adopted: a personalized DTW template library is established through the initial 5-8 shots, and joint angular velocity difference features are introduced. Enhance temporal sensitivity and calibrate spatial perception biases (such as position angle offsets) in the environment.

[0211] Step S34: Identify and assess the quality of the action.

[0212] In this embodiment, based on the processed and normalized feature sequence This forms the network input. The slow path is mainly used for spatial semantic extraction and temporal step size. (Sample 1 frame every 16 frames), input frame rate = 8fps, using 3DResNet-50 architecture, number of channels = 2048, extract spatial features (such as the forward lean posture of the torso in a forehand attack and the wrist pronation shape in a backhand topspin) through 4 layers of residual blocks, and output slow path features. :

[0213]

[0214] in, Represent a real-valued matrix, Represents the normalized feature sequence Total number of time steps Corresponding to the time dimension, it represents the number of time steps of the Slow path output features, that is, the number of frames retained after time step sampling; The corresponding channel dimension represents the feature dimension of each time step.

[0215] Fast paths are primarily used for fast motion capture, with a time step size of [missing information]. ( =8, frame rate = 64fps), no time downsampling is performed throughout, a lightweight 3D convolutional structure is used, the number of channels = 256 (1 / 8 of the Slow path), and the output is Fast path features. :

[0216]

[0217] Perform time-dimensional alignment, Using time-stride convolution (stride=8, kernel size=8), the temporal resolution is increased from... Down to ,get To unify the channel dimensions, Performing a 1×1×1 convolution increases the number of channels from 256 to 2048, and... The number of channels is consistent; then feature fusion is performed, and the fusion weights are calculated using a gated recurrent unit (GRU). :

[0218]

[0219] in, This represents the learnable weight matrix used to compute the fusion weights in a gated recurrent unit (GRU). This represents the learnable bias parameters corresponding to the weight matrix.

[0220] Output fusion features :

[0221]

[0222] Will Input a fully connected layer and a Softmax activation function, and output the classification probabilities of table tennis technique movements and movement quality.

[0223] The method of this invention has significant theoretical advantages in terms of algorithm complexity and model design. Through multi-stage feature fusion and dynamic time warping mechanisms, this system can achieve accurate recognition and quality assessment of table tennis movements in real-time video streams (≥30fps). Compared with traditional single-viewpoint or sensor-based solutions, this system has significant potential for improvement in detection accuracy and robustness. During implementation, the system has undergone functional verification and can stably output table tennis hitting events and movement quality assessment results, with a system response time of less than 100ms, demonstrating real-time performance and stability.

[0224] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0225] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A table tennis motion detection system based on pure visual perception, characterized in that: It includes a table tennis ball detection and event monitoring module, a human-racket posture estimation module, and a temporal motion recognition and quality assessment module; The table tennis ball detection and event monitoring module is used to receive multi-view video streams and output the motion trajectory of the table tennis ball and the sequence of hitting events; The table tennis ball detection and event monitoring module adopts an improved DTTNet model based on deep learning. It detects the trajectory of the table tennis ball through multi-frame video stacking and deformable convolution, and identifies events such as touching the table, touching the net, and crossing the net. The human-racket posture estimation module is used to receive the multi-view video stream and output the coordinates of the user's human body key points and the coordinates of the racket key points; The human-racket posture estimation module adopts a joint inference structure of lightweight detection network and high-resolution posture estimation network, outputs the coordinates of human key points and racket three key points, and constructs a joint kinematic model of "human-racket". The human-racket pose estimation module also includes: training using a dedicated dataset supplemented with annotations for three key racket points—the handle start point, the grip end, and the racket face center—to uniquely represent the racket's spatial pose; and defining skeletal connectivity relationships for table tennis motion analysis, including: Head pose connection: Connecting key points between the eyes and ears forms the basis for head pose estimation; Trunk and limb connection: connecting key points such as the shoulder, elbow, wrist, hip, knee, and ankle to form the kinetic chain of the main limbs of the human body; Human-racket association connection: connecting key points on the wrist with key points at the starting point of the racket handle to establish an interactive relationship between the user and the equipment; Racket vector axis connection: Sequentially connect the starting point of the handle, the end of the grip, and the key points at the center of the racket face to construct a racket reference system for calculating the racket face orientation and swing trajectory; The temporal action recognition and quality assessment module is used to fuse the ball hitting event sequence, human body key point coordinates and racket key point coordinates to output the user's technical action classification and quality score. The temporal action recognition and quality assessment module integrates multi-view video and posture features, and classifies and assesses the quality of table tennis technical actions through temporal boundary matching and multi-scale feature analysis. The temporal action recognition and quality assessment module includes: receiving multi-view video streams using a boundary matching network and outputting the start and end frames of the technical actions; extracting and outputting the user's core joint coordinates and kinematic parameters through key body features; and using a SlowFast dual-path architecture to fuse key body features and video spatiotemporal features to output the classification results and action quality assessment results of table tennis technical actions, covering the three major technical systems of forehand, backhand, and chop.

2. The table tennis motion detection system based on pure visual perception according to claim 1, characterized in that: The improved DTTNet model based on deep learning includes: through multi-frame spatiotemporal joint modeling and a unified two-stage processing mechanism, the generated features are highly shared and reused in parallel to achieve the localization and tracking of ping-pong balls and the monitoring of key events.

3. The table tennis motion detection system based on pure visual perception according to claim 2, characterized in that: The implementation methods of the table tennis ball detection and event monitoring module include: Step S11: Perform feature processing using the spatiotemporal context information of the target frame and its preceding and following frames to construct the input tensor; Step S12: Extract low-level features based on the input tensor, generate globally shared features, and obtain preliminary coordinates of the coarse localization result; Step S13: Based on the preliminary coordinates of the coarse localization results, the original image is locally cropped. The cropped image is then standardized and used as the input to the network for refined detection, outputting accurate shared features. Step S14: Use the globally shared features and the precisely shared features as input features for table tennis detection and table tennis key event monitoring, and output fine coordinates; Step S15: Calculate the ping-pong ball detection position in the original image by mapping the preliminary coordinates and the fine coordinates according to their positional relationship; Step S16: By reusing the refined detection output of precise shared features, extract local refined features, and splice and fuse them with global event features as the final input of the classification head, the video frame is classified into four categories of table tennis events, including table tennis bounce, net touch, net over, and empty events.

4. The table tennis motion detection system based on pure visual perception according to claim 3, characterized in that: The monitoring of key events in table tennis also includes: using an event recognition model trained on the publicly available OpenTTGames dataset, employing a weighted cross-entropy loss function during training to address class imbalance, and introducing a smooth event labeling strategy. When constructing the labels for the training dataset, for the manually labeled center frame of an event, the probability weight of the event is propagated to the four frames before and after it, so that the weight of the true label in adjacent frames decays smoothly, rather than being set as a hard label. The value of the decay weight is calculated based on the form of a sine function.

5. The table tennis motion detection system based on pure visual perception according to claim 4, characterized in that: The human-racket joint pose estimation module includes: A joint inference architecture is constructed, which includes the PicoDet object detection model and the HRNet keypoint detection model. The PicoDet object detection model includes: an ESNet backbone network, which is used to extract multi-scale features step by step through depthwise separable convolution and channel shuffling; the HRNet model adopts a multi-resolution parallel branch architecture, which repeatedly performs feature exchange and fusion on sub-networks of different resolutions, outputs features from the highest resolution branch and generates a keypoint heatmap. The user's location is determined using a lightweight object detection network; The key point coordinates of the human body and racket are regressed using a high-resolution convolutional neural network.

6. The table tennis motion detection system based on pure visual perception according to claim 5, characterized in that: The implementation methods of the human-racket pose estimation module include: Step S21: Extract initial feature maps using the input video frames, and then extract features at different scales based on the initial feature maps; Step S22: Use CSP-PAN feature pyramid to fuse features extracted at different scales and output a multi-level feature map with a unified channel dimension; Step S23: Predict the user's bounding box based on multi-level feature maps; Step S24: Use the predicted bounding box to crop the target region from the original input to form a standardized input block; Step S25: Perform multi-resolution feature fusion based on standardized input blocks to generate key points; Step S26: Decode the generated key points to generate standardized structured pose information.

7. A table tennis motion detection system based on pure visual perception according to claim 6, characterized in that: The implementation methods of the timing action recognition and quality assessment module include: Step S31: Using the user region image patch after being cropped by the bounding box output by the human-racket pose estimation module in the multi-view video stream as input, a pre-trained dual-stream network is used to extract the spatiotemporal joint feature sequence; Step S32: Through convolution and the Sigmoid activation function, predict the probability sequence of each time point as the start / end of the action in real time, generate initial action proposals using a sliding window, and screen the quality of the initial proposals through boundary matching sampling and double confidence calculation; Step S33: Establish a personalized baseline coordinate system and extract key body features; Step S34: Identify and assess the quality of the actions, and output high-scoring action sequence segments.

Images