A table tennis trajectory detection system based on adjacent frame feature multiplexing network

Abstract

The present invention relates to a table tennis trajectory detection system based on an adjacent frame feature multiplexing network. The system comprises a main control device, a horizontal camera, a top-down camera, and a video data processing module. The camera is used to collect table tennis video data in the horizontal direction and from a top-down perspective of the table tennis table, and transmit the collected video data to the main control device for processing. The video data processing module is used to transmit video data to the main control device and perform trajectory record analysis. The main control device is equipped with an adjacent frame feature multiplexing network, which detects table tennis trajectory based on the table tennis video data collected by the camera. The adjacent frame feature multiplexing network includes a feature storage and return module that can store the output of the network layer at the current moment and transmit the features to the input end of the network layer at the next moment, thereby achieving efficient feature reuse. Compared with the existing technology, the present invention has the advantages of high detection accuracy, accurate target positioning, and small model parameters.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a table tennis trajectory detection system based on an adjacent frame feature multiplexing network. Background Art

[0002] Real-time recording of the ball's trajectory during a table tennis match can help analyze the opponent's offensive characteristics and weaknesses. Currently, ball trajectory analysis relies mainly on human observation and lacks certain data theory support.

[0003] Applying artificial intelligence technology to table tennis trajectory detection is a new research direction. It not only helps save labor costs, but also improves detection accuracy and provides rich theoretical support. For example, Chinese patent application CN115457083A discloses a table tennis motion parameter detection method, system, and device, belonging to the field of table tennis technology. The method includes the following steps: capturing images of a table tennis ball in motion using a first camera positioned above the table tennis table and a second camera positioned above the side of the table tennis table; constructing a three-dimensional coordinate system and obtaining the coordinate values ​​of the table tennis ball; obtaining the motion trajectory of the table tennis ball based on the coordinate values; determining whether the table tennis ball hits a rebound plate, and if so, calculating the angle at which the table tennis ball hits the rebound plate; presetting multiple reference points, selecting several frames of images of the table tennis ball passing through the reference points, and obtaining the time interval between the images based on the pre-set reference points; and calculating the average velocity of the table tennis ball during its motion based on the pre-set reference points and time interval. This application is applicable to table tennis tables for single-player training and can collect motion parameters of the ball during its motion, with the advantages of good tracking precision and high accuracy. The defects of this application are: the network model structure is complex, resulting in slow calculation speed; the benchmark points need to be preset manually, and the initial work cost is high. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a table tennis trajectory detection system based on an adjacent frame feature multiplexing network.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A table tennis trajectory detection system based on an adjacent frame feature multiplexing network includes a main control device, a horizontal camera, a top-down camera, a video data processing module, and a power supply module;

[0007] The power module is used to supply power to the main control device; the main control device is connected to the horizontal camera, the top view camera and the video data processing module respectively;

[0008] The horizontal camera is used to collect table tennis video data in the horizontal direction and transmit the collected video data to the main control device for processing;

[0009] The overhead camera is used to collect table tennis video data from a top-down perspective of the table tennis table and transmit the collected video data to the main control device for processing;

[0010] The video data processing module is used to transmit video data to the main control device and perform trajectory record analysis;

[0011] The main control device is equipped with an adjacent frame feature multiplexing network, which detects the trajectory of the table tennis ball based on the table tennis video data collected by the camera;

[0012] The adjacent frame feature multiplexing network includes a feature storage and return module, which can store the output of the network layer at the current moment and pass the feature to the input end of the network layer at the next moment to achieve efficient feature multiplexing.

[0013] Furthermore, the adjacent frame feature multiplexing network includes six network layers for feature extraction.

[0014] Furthermore, after each layer of the adjacent frame feature multiplexing network completes the feature extraction work, the extracted features are passed to the next network layer, and at the same time, are passed to the feature storage and return module to save the feature information extracted at the current moment;

[0015] The input of the first layer of the network is the table tennis video frame data, and the output is the extracted features. The input of each subsequent layer of the network is the output of the upper layer of the network and the feature information of the previous moment saved in the feature storage feedback module.

[0016] Furthermore, the feature storage and return module is used to realize feature reuse. The feature storage and return module includes a Transformer model. The Transformer model is integrated with the TokenLearner mechanism, which only converts and generates a small number of tokens from the feature map to reduce the computational complexity of the Transformer model.

[0017] Furthermore, the calculation process of the TokenLearner mechanism includes the following steps:

[0018] Note that the size of the feature map input to the Transformer module is x∈R H×W×C , using the TokenLearner mechanism, N groups of spatial attention mechanisms are used to calculate the corresponding spatial attention weight heat map with the input feature map, and then the weight heat map is multiplied with the input feature map to complete the assignment operation, and N weighted feature vectors z∈R are obtained. H ×W×C , the calculation process is expressed as:

[0019]

[0020] Among them, x represents the input feature map, α i (x) represents the i-th attention weight, W i represents the i-th weight matrix;

[0021] Use spatial pooling to transform the weighted z∈R H×W×C Transformed into z′∈R 1×1×C , which are tokens.

[0022] Furthermore, each Transformer module sets up six groups of spatial attention mechanism extraction networks, that is, the number of tokens input to each Transformer module is 4.

[0023] Furthermore, when the feature storage and feedback module transmits the features of the previous moment back to the current moment, it first uses a 3x3 convolutional layer to further extract the features, and then uses a Transformer module to obtain global feature information.

[0024] Furthermore, after the feature storage and feedback module uses the Transformer module to obtain global feature information, a dropout layer is used when returning the global feature information to the backbone network to prevent overfitting due to the high feature similarity in the frame sequence.

[0025] Furthermore, the detector of the adjacent frame feature multiplexing network includes a classifier and a position regressor for determining the positive and negative categories and position information of the target;

[0026] The target position information detected in the current frame of the table tennis video data is stored in the adjacent frame feature multiplexing network. When detecting the next frame, the stored position information is predicted and corrected through the Kalman filter, and the obtained position is mapped in the feature map and passed to the position regressor to predict the target of the current frame;

[0027] For the returned location information parameters, use Indicates that (u, v) represents the center point of the target at the previous moment, h is the height of the position prediction frame at the previous moment, and γ is the aspect ratio of the position prediction frame at the previous moment; The velocity components are represented as (u, v, γ, h) respectively, and the Kalman filter of the uniform velocity model and the linear observation model is used to predict (u, v, γ, h) at the current moment.

[0028] Furthermore, a position coordinate storage module is configured to store the position information output by the position regressor. When determining the position at the next moment, the stored position coordinate information will be predicted by the Kalman filter, and the predicted position will be mapped to the feature map. The regressor will make more accurate position judgments based on the position information.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention achieves network lightweighting and improved detection accuracy by increasing the feature reuse rate. The core of the network is the feature storage and return module, which can store the output of the network layer at the current moment and pass the feature to the input end of the network layer at the next moment to achieve efficient feature reuse. The Transformer model is used in this module to perform secondary processing on the features, construct global correlation information, and improve the feature richness of the feature map. According to experimental tests, the network has a detection accuracy of up to 96.8% for table tennis and a positioning accuracy of up to 89.1% for targets. The parameter size of the model is only 7.68MB, which is highly practical.

[0031] 2. By adding a feature storage and feedback module, the present invention enables the feature extraction network to fully utilize feature information. As the number of detections increases, the stored and reused feature information becomes richer, that is, the extraction effect of the deep network is achieved through the superposition and fusion of features in the time dimension, and high detection accuracy is achieved with smaller model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a diagram of the system hardware structure design of the present invention;

[0033] Figure 2 Schematic diagram of a collection scenario in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the timing feature multiplexing network structure of the present invention;

[0035] Figure 4 Schematic diagram of the calculation process of the TokenLearner mechanism in an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of the structure of a feature storage and transmission module in an embodiment of the present invention;

[0037] Figure 6 A schematic diagram of a detailed network structure in an embodiment of the present invention;

[0038] Figure 7 This is a diagram of an experimental data set in an embodiment of the present invention;

[0039] Figure 8: is the x-coordinate trajectory diagram of the spherical motion in an embodiment of the present invention;

[0040] Figure 9 y coordinate trajectory diagram of the spherical motion in an embodiment of the present invention;

[0041] Figure 10 Graph showing the z-coordinate trajectory of the sphere's motion in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0043] Real-time recording of the ball's trajectory during a table tennis match can help analyze the opponent's offensive characteristics and weaknesses. Currently, ball path analysis relies primarily on human observation, lacking theoretical support. Therefore, this paper proposes a table tennis trajectory extraction network based on a target detection algorithm to record the trajectory of a table tennis ball in a video. Two 120FPS high-definition cameras are used to capture table tennis trajectory videos from horizontal and top-down angles, respectively. A Raspberry Pi 4B serves as the transmission device, receiving video information via a USB port. This information is then transmitted to a PC using a Wi-Fi module for video data processing and trajectory analysis.

[0044] The table tennis trajectory extraction network based on the target detection algorithm achieves network lightweighting and improved detection accuracy by increasing the feature reuse rate. The core of the network is the "feature storage and return" module, which can store the output of the network layer at the current moment and pass the feature to the input of the network layer at the next moment to achieve efficient feature reuse. The Transformer model is used in this module to perform secondary processing on the features, construct global correlation information, and improve the feature richness of the feature map. Experimental tests have shown that the network has a detection accuracy of 96.8% for table tennis and a positioning accuracy of 89.1% for targets. The parameter size of the model is only 7.68MB, which is highly practical.

[0045] System hardware structure design

[0046] Specifically, such as Figure 1 As shown in the figure, it is the system hardware structure design diagram of the present invention. Since the use of wired signal transmission may cause safety hazards in the actual application scenario of the system, wireless communication is used to transmit the collected video signal to the PC terminal device for the next step of processing. Figure 1As shown in the figure, two 120FPS high-definition cameras are connected to the Raspberry Pi 4B through the USB 3.0 interface. The USB-C power adapter provides a stable 5V voltage to the entire hardware system. The Raspberry Pi 4B network is configured to connect to the wireless network. At the same time, the PC is connected to the same network as the Raspberry Pi 4B to complete the hardware foundation of signal transmission.

[0047] In the hardware solution of this system, the Raspberry Pi 4B is used as the main hardware device for communication transmission. It is equipped with a quad-core Cortex-A72 (ARM v8) 64-bit processor, which can provide better data processing performance to meet the needs of high-speed transmission. It has a built-in dual-band 2.4GHz and 5.0GHz wireless network card and supports IEEE 802.11b / g / n / ac standards, enabling wireless connection with the PC to complete video data transmission. At the same time, the Raspberry Pi 4B is equipped with a USB 3.0 interface, which can achieve high-speed data transmission with the HD camera, playing a key role in the transmission of table tennis video data captured by the 120FPS HD camera. The relevant parameter information of the Raspberry Pi 4B is shown in Table 1 below:

[0048] Table 1 Raspberry Pi 4B related parameters

[0049]

[0050] Based on this communication control hardware, a wireless local area network (WLAN) mode is used. The Raspberry Pi 4B and the PC are connected to the same wireless router to form a wireless local area network. In LAN mode, communication between the Raspberry Pi 4B and the PC is based on the TCP / IP protocol stack and is in the same subnet, thus completing the communication relationship.

[0051] The trajectory detection network described in this article requires a high frame rate for video information. Therefore, the hardware system uses the E-Con Systems See3CAM_130 high-frame-rate camera, compatible with the Raspberry Pi 4B, as the information acquisition device. This camera supports up to 120 FPS video streaming at 720p resolution, providing high-quality images and a high dynamic range. It also has excellent low-light performance, providing high-quality video images even in low-light conditions, meeting the needs of various scenarios. The relevant hardware parameters are shown in Table 2 below:

[0052] Table 2 E-Con Systems See3CAM_130 related parameters

[0053]

[0054] For hardware power, I used the official Raspberry Pi USB-C power adapter, which delivers a stable 5.1V / 3A current. This ensured the Raspberry Pi 4B received sufficient power for normal operation even when connected to multiple peripherals (such as the See3CAM_130 camera). The E-Con Systems See3CAM_130 directly draws power from the Raspberry Pi 4B motherboard via USB, eliminating the need for a separate battery.

[0055] Based on the above hardware foundation, the video information collection requirements of the table tennis trajectory detection network in this paper have been met. Figure 2 The collection scene is shown as follows.

[0056] Network main structure design

[0057] In the main structure of the network in this paper, the utilization rate of features is improved to reduce the use of network layers. Therefore, in the network design of this paper, the features of the previous moment are transferred and reused to the current moment. According to this feature, the network is named the adjacent frame feature reuse network. The network structure is as follows Figure 3 shown.

[0058] like Figure 3 As shown in the figure, the entire network uses a total of 6 network layers. For the output of each network layer, after it completes the feature extraction work, the extracted features are not only passed to the next network layer, but also passed to the feature storage and return module to save the feature information extracted at the current moment. For the input of the network layer, in addition to the conventional upper-layer network transmission output, it also includes the feature information of the previous moment saved in the feature storage and return module. After adding the storage module, the feature extraction network can make full use of the feature information, and as the number of detections increases, the stored and reused feature information becomes richer, that is, the extraction effect of the deep network is achieved through the superposition and fusion of features in the time dimension.

[0059] Feature storage and return module design

[0060] Since the number of network layers in the backbone network is small and only three downsampling operations are performed, the feature information extracted by the convolutional network is mainly local feature information. In order to achieve effective feature reuse, this paper adds a Transformer model to the feature storage and feedback module to perform global attention calculations on the returned features, and construct long-distance feature relationships of the features in the feature map to functionally supplement the convolutional network. However, due to the large amount of computation required by the Transformer model, adding a Transformer model to each module will cause the number of network parameters to surge, losing the advantage of being lightweight. Therefore, this paper integrates the TokenLearner mechanism into the Transformer model, and only generates a small number of tokens from the feature map to reduce the amount of computation required by the Transformer. The calculation process of the TokenLearner mechanism used is as follows: Figure 4 shown.

[0061] The feature map size input to the Transformer module is x∈R H×W×C , which is different from the operation of dividing the feature map into feature blocks and converting them into one-dimensional tokens, the TokenLearner mechanism uses N groups of spatial attention mechanisms to calculate the corresponding spatial attention weight heat map with the input feature map, and then multiplies the weight heat map with the input to complete the assignment operation to obtain N weighted feature vectors z∈R H×W×C , the calculation process can be expressed as:

[0062]

[0063] Among them, x represents the input feature map, W i Represents the i-th weight matrix, which is used to transform the input feature map x into a specific output feature map z i The weight matrix is ​​a key component of a neural network, used to learn the mapping relationship from input to output; α i (x) represents the i-th attention weight. In some calculations, we may want to weight the input feature map x according to some attention mechanism in order to focus on a specific part or area. i These weights can be calculated based on self-attention or other attention methods. Applying these weights to the input feature map x can obtain the weighted output feature map z i .

[0064] Then spatial pooling is used to transform the weighted z∈R H×W×C Transformed into z′∈R 1×1×C, which are tokens. Compared with the standard token acquisition method, TokenLearner can determine the number of generated tokens by setting the number of spatial attention calculation groups, avoiding the situation where the number of tokens generated by feature map division is too large and the calculation sequence is too long. And because of the use of the spatial attention mechanism, the model can adaptively extract the important areas of the image, and unlike the disadvantage of poor data correlation caused by dividing the feature map, the feature map can be directly converted into tokens through spatial pooling without segmentation, which effectively ensures the consistency of feature information. In this paper, six groups of spatial attention mechanism extraction networks are set for each Transformer module, that is, the number of tokens input to each Transformer module is 4. The detailed structure of the feature storage and feedback module is as follows. Figure 5 shown.

[0065] Feature storage and return module

[0066] When passing the features of the previous moment back to the current moment, a 3x3 convolutional layer is first used to further extract the features, and then a Transformer module is used to obtain global feature information. Finally, when passing the features back to the backbone network, a dropout layer is used to prevent overfitting due to the high similarity of features in the frame sequence.

[0067] Location information feedback module design

[0068] The network's detector consists of a classifier and a position regressor. The trajectory of the ping-pong ball in the video is continuous and regular. That is, after extracting frames from the video, the position of the ping-pong ball in the current frame is only slightly offset from that in the previous frame (the camera is fixed).

[0069] Therefore, the target position information detected in the current frame is stored in the network. When the next frame is detected, the stored position information is predicted and corrected through the Kalman filter, and the obtained position is mapped in the feature map and passed to the regressor to predict the target of the current frame.

[0070] For the returned location information parameters, use Indicates that (u, v) represents the center point of the target at the previous moment, h is the height of the position prediction frame at the previous moment, and γ is the aspect ratio of the position prediction frame at the previous moment; It is represented by the velocity components of the above four elements, and the Kalman filter of the uniform velocity model and the linear observation model is used to predict the current moment (u, v, γ, h);

[0071] In summary, the unit network structure diagram after the combination of each optimization module is as follows Figure 6 shown.

[0072] Detailed network structure

[0073] like Figure 6 Taking the last dimension module as an example, after the convolution module 2 receives the feature map output by the convolution module 1, it completes the feature extraction operation and transmits the feature to the storage unit to retain the extracted feature information. When detecting the frame at the next moment, the stored feature is used by the lightweight Transformer module to construct a global feature relationship to improve the visibility of the target in the picture. In this operation, the feature map of the previous moment stored is first subjected to a feature transition operation by a 3x3 convolution, and then four spatial attention submodules are set up to calculate the relevant heat map. The calculated spatial heat weight map is assigned to the feature map to obtain four sub-feature maps and converted into tokens for multi-head self-attention calculation. Finally, the obtained feature map is fused with the input of the convolution module 1 to improve the richness of the input information.

[0074] If convolution module 2 is the last feature extraction module, the features it outputs will be passed to the classifier and regressor to determine the positive and negative categories and location information of the target. In addition, the location information output by the regressor is also stored in a location coordinate storage module. When the position is determined at the next moment, the stored location coordinate information will be predicted by the Kalman filter and the predicted position will be mapped to the feature map. The regressor can then make a more accurate location judgment based on this location information.

[0075] Dataset creation and experimental environment introduction

[0076] Since the position information obtained by the detection is relative to the image rather than the actual position information, this paper obtains the position information of the ball relative to the table by shooting horizontally and vertically relative to the table, so that the real position information can be obtained through later conversion. Based on this principle, this paper shoots 50 5-minute table tennis match videos from two angles and creates a data set by extracting frames, as shown in the figure. Figure 7 shown.

[0077] The experimental environment and network hyperparameter settings used in this experiment are as follows: The hardware environment uses an Intel i7 11700 CPU and an Nvidia RTX 3070 GPU with 5888 CUDA cores and 184 Tensor cores. For software, PyTorch 1.11.0 was used as the deep learning framework API, along with the CUDA computing platform version 11.3 and the cuDNN computing acceleration library version 8.2.1. The following hyperparameter settings were obtained after multiple training runs for the key hyperparameters in the experimental environment: 200 epochs, an initial learning rate of 0.001 that gradually converged to 0.00001 during training, a momentum parameter of 0.954, and a weight decay rate of 0.0013.

[0078] This article evaluates the performance of the network model from three aspects: recognition accuracy, positioning accuracy, and model complexity. The following objective indicators are used to analyze the model:

[0079] Recognition accuracy:

[0080]

[0081] Recognition recall rate:

[0082]

[0083] Recognition accuracy:

[0084]

[0085] Positioning accuracy:

[0086]

[0087] Model complexity:

[0088] Parameter=(C in ×K 2 +1)C out

[0089] (*Note: TP is the number of positive classes predicted as positive classes; FP is the number of negative classes predicted as positive classes; FN is the number of positive classes predicted as negative classes; GroundTruthBox is the marked location box; PredictionBox is the location box predicted by the model output; C in 、C out are the number of channels before and after the feature map input; K is the size of the convolution kernel).

[0090] Ablation experiments

[0091] In the experimental phase, this example first used an ablation experiment to decompose the network model in this paper into six networks. Through data comparison, the effectiveness of each network module designed in this paper in the overall network was determined. In the experiment, five different sets of hyperparameters were set to train the six models in the ablation experiment multiple times, and the average value was taken as the optimal parameter performance index value for each model. The set hyperparameter control group is as follows:

[0092] Table 1 Hyperparameter settings

[0093] Epoch Learning rate (LR) momentum Weight_Decay 1 300 0.0001 0.9 0.0005 2 300 0.0001 0.95 0.0005 3 500 0.001 0.9 0.0005 4 500 0.0001 0.95 0.0005 5 700 0.001 0.9 0.0005 6 700 0.0001 0.95 0.0005

[0094] Taking all of the aforementioned training data into account, after adding the backhaul module and the Transformer feature secondary processing module, the model can be trained with only 500 iterations. Excessive iterations can cause network oscillation during subsequent training, resulting in a certain decrease in accuracy. For a backbone network consisting solely of residual modules, due to its simple structure and weak feature learning ability, more iterations are required to ensure model convergence. In summary, the optimal evaluation parameters for each network model were extracted for comparative testing.

[0095] Table 2. Comparison results of ablation experiments

[0096]

[0097] The data comparison between Network 1 and Network 2 primarily evaluates whether the addition of a feature storage unit improves network performance. Network 1 includes only a backbone network, with only six convolutional layers for feature extraction. Due to the inability to fully extract feature information, the model performs poorly in both recognition and localization accuracy, and the recall rate is low, indicating missed detections and an inability to fully capture the target's trajectory coordinates. Poor localization accuracy also renders the captured coordinate information unreliable. Network 2 incorporates a storage module to reuse feature information extracted from the previous frame, increasing the network's depth in the temporal dimension. This enriches the feature information extracted as the number of detection frames increases, significantly improving recognition and localization accuracy with only a slight increase in network complexity. Both Networks 3 and 4 incorporate a Transformer model in their backhaul modules. The difference between the two is that Network 3 uses a standard Transformer module, while Network 4 uses a lightweight Transformer module. Comparing Network 3 and Network 4 with Network 2 shows that the addition of the Transformer module performs secondary processing on stored features, extracting global long-range dependency features that complement the local features in the backbone network, effectively improving network performance. However, the addition of the Transformer module increases the network complexity of Networks 3 and 4 compared to Network 2. However, comparing the complexity of Network 3 and Network 4, Network 4, due to its lightweight Transformer module design, is only half the complexity of Network 3 and slightly exceeds Network 3 in detection accuracy. Networks 5 and 6 both incorporate a Kalman filter in the position information feedback, predicting the position of the sphere in the current frame based on the sphere's trajectory in the previous frame. Comparisons of these two networks with Networks 3 and 4, respectively, show that the addition of the Kalman filter effectively improves the accuracy of position determination.

[0098] A comparison shows that the six network models have significant differences in their results for determining target locations. The location area predicted by Network 1 deviates significantly from the actual location of the target. Network 2, with the addition of a feedback module, can roughly select the target. Networks 3 and 4, after adding a transformer model for secondary processing, further improve detection accuracy compared to Network 2. In Networks 5 and 6, Kalman filters are added to determine location, and the predicted output location information is highly consistent with GroundTruth.

[0099] Secondly, the position information obtained after horizontal and vertical shooting is spatially modeled to determine the trajectory of the sphere. In this control experiment, the Ground Truth sample is obtained by shooting with a depth camera, and the rest of the motion trajectories are obtained using the method described in this article. The comparison is as follows: Figure 8 shown.

[0100] from Figures 8-10 The four figures in Figure 1 intuitively demonstrate the accuracy of each model in detecting the sphere's trajectory during the ablation experiment. The depiction of the trajectory in both the overall dimension and the various coordinate dimensions agrees with the ablation experiment results: the optimized network model 6 is able to more accurately detect the sphere's position, and the resulting trajectory map is more closely aligned with the sphere's true trajectory.

[0101] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A table tennis trajectory detection system based on adjacent frame feature multiplexing network, characterized in that: Including main control equipment, horizontal camera, overhead camera, video data processing module and power module; The power module is used to supply power to the main control device; the main control device is connected to the horizontal camera, the top view camera and the video data processing module respectively; The horizontal camera is used to collect table tennis video data in the horizontal direction and transmit the collected video data to the main control device for processing; The overhead camera is used to collect table tennis video data from a top-down perspective of the table tennis table and transmit the collected video data to the main control device for processing; The video data processing module is used to transmit video data to the main control device and perform trajectory record analysis; The main control device is equipped with an adjacent frame feature multiplexing network, which detects the trajectory of the table tennis ball based on the table tennis video data collected by the camera; The adjacent frame feature multiplexing network includes a feature storage and return module, which can store the output of the network layer at the current moment and pass the feature to the input end of the network layer at the next moment to achieve efficient feature multiplexing; The detector of the adjacent frame feature multiplexing network includes a classifier and a position regressor, which are used to determine the positive and negative categories and position information of the target; The target position information detected in the current frame of the table tennis video data is stored in the adjacent frame feature multiplexing network. When detecting the next frame, the stored position information is predicted and corrected through the Kalman filter, and the obtained position is mapped in the feature map and passed to the position regressor to predict the target of the current frame; For the returned location information parameters, use Indicates that (u, v) represents the center point of the target at the previous moment, h is the height of the position prediction frame at the previous moment, and γ is the aspect ratio of the position prediction frame at the previous moment; The velocity components are represented as (u, v, γ, h) respectively, and the Kalman filter of the uniform velocity model and the linear observation model is used to predict the current moment (u, v, γ, h); The position information output by the position regressor is stored in a position coordinate storage module. When the position is determined at the next moment, the stored position coordinate information will be predicted by the Kalman filter, and the predicted position will be mapped to the feature map. The regressor can make more accurate position judgments based on this position information.

2. A table tennis trajectory detection system based on adjacent frame feature multiplexing network according to claim 1, characterized in that: The adjacent frame feature multiplexing network includes six network layers for feature extraction.

3. The table tennis trajectory detection system based on adjacent frame feature multiplexing network according to claim 1, characterized in that: After each layer of the adjacent frame feature multiplexing network completes the feature extraction work, the extracted features are passed to the next network layer, and at the same time passed to the feature storage and return module to save the feature information extracted at the current moment; The input of the first layer of the network is the table tennis video frame data, and the output is the extracted features. The input of each subsequent layer of the network is the output of the upper layer of the network and the feature information of the previous moment saved in the feature storage feedback module.

4. The table tennis trajectory detection system based on adjacent frame feature multiplexing network according to claim 1, characterized in that: The feature storage and return module is used to realize feature reuse. The feature storage and return module includes a Transformer model. The Transformer model is integrated with the TokenLearner mechanism, which only generates a small number of tokens from the feature map to reduce the computational complexity of the Transformer model.

5. A table tennis trajectory detection system based on adjacent frame feature multiplexing network according to claim 4, characterized in that: The calculation process of the TokenLearner mechanism includes the following steps: Note that the size of the feature map input to the Transformer module is x∈R H×W×C , using the TokenLearner mechanism, N groups of spatial attention mechanisms are used to calculate the corresponding spatial attention weight heat map with the input feature map, and then the weight heat map is multiplied with the input feature map to complete the assignment operation, and N weighted feature vectors z∈R are obtained. H×W×C , the calculation process is expressed as: Among them, x represents the input feature map, α i (x) represents the i-th attention weight, W i represents the i-th weight matrix; Use spatial pooling to transform the weighted z∈R H×W×C Transformed into z′∈R 1×1×C , which are tokens.

6. The table tennis trajectory detection system based on adjacent frame feature multiplexing network according to claim 4, characterized in that: Each Transformer module sets up six groups of spatial attention mechanism extraction networks, that is, the number of tokens input to each Transformer module is 4.

7. The table tennis trajectory detection system based on adjacent frame feature multiplexing network according to claim 1, characterized in that: When the feature storage and return module returns the features of the previous moment to the current moment, it first uses a 3x3 convolutional layer to further extract the features, and then uses a Transformer module to obtain global feature information.

8. The table tennis trajectory detection system based on adjacent frame feature multiplexing network according to claim 1, characterized in that: After the feature storage and return module uses the Transformer module to obtain global feature information, a dropout layer is used when returning the global feature information to the backbone network to prevent overfitting due to the high feature similarity in the frame sequence.

Images