A table tennis biomechanical fingerprint visual analysis system based on attention mechanism

Abstract

The invention discloses a table tennis biomechanical fingerprint visual analysis system based on an attention mechanism, comprising a data reorganization module, a data generation module, a data fusion module and a fingerprint visualization module; the data reorganization module is used to transform service data in table tennis into three forms: spatial form, motion form and time form; the feature generation module is used to generate overall features, spatial features, kinematic features and time features according to the data after the form transformation by using a model constructed based on the attention mechanism; the feature fusion module is used to integrate all features generated by the feature generation module into a biomechanical fingerprint for technical identification; the fingerprint visualization module is used to visualize the biomechanical fingerprint and support visualized interactive analysis, and the system can realize visual analysis of table tennis biomechanical fingerprints.

Description

Technical Field

[0001] The invention belongs to the technical field of table tennis visualization, and in particular relates to a table tennis biomechanical fingerprint visual analysis system based on an attention mechanism. Background Art

[0002] Table tennis is a challenging and competitive sport in which players use a variety of techniques to counter their opponent's shots. The success of these techniques depends largely on the player's physical coordination. For example, when hitting a common forehand loop, the speed and rotation of the ball are affected by the speed and angle of rotation of the player's torso and shoulders. In order to help players optimize their physical coordination for different techniques, researchers have conducted extensive research to explore the biomechanics of table tennis. Traditionally, researchers use sensors or motion capture devices with inertial measurement units (IMUs) to collect players' biomechanical data, including acceleration, angle, angular velocity, etc. Through this data, these studies focus on the biomechanical mechanisms of players when hitting the ball and the biomechanical differences between different players when using different techniques. However, few studies have investigated the biomechanical fingerprint of individual players.

[0003] Biomechanical research in table tennis aims to investigate the mechanical properties of a player's body while using various techniques. For example, Iino and Kojima explored the energy generation mechanisms of the upper extremities when executing a topspin forehand. Building on these studies, Iino et al. found that the lower extremities have a significant influence on the performance of the upper extremities, leading to an extensive investigation of the role of the lower extremities in the execution of a topspin forehand. In addition, researchers have explored the biomechanical properties of other body parts, including the trunk and soles. A comprehensive review of these studies can be found at . Insights from these investigations can help coaches develop training programs to improve player performance.

[0004] However, these studies have attempted to discover common features by examining the biomechanical data of multiple players. The specific characteristics of individual players, also referred to in this work as biomechanical fingerprints, have not been fully investigated. Gravenhorst et al. selected 177 performance indicators as fingerprints of rowers to characterize their technical style. Although this approach can help coaches find the best rowers for rowing, it is time-consuming because researchers have to examine quite a large number of indicators. In addition, these indicators are not compatible with other sports.

[0005] In recent years, there are more and more visualization methods for sports analysis. Time series visualization is often used to show the spatiotemporal characteristics of a game. For example, line graphs and bar graphs are popular choices for showing the changes in score-related statistics and player performance indicators in football and basketball games. Trajectories are also commonly used to show the movements of players. In addition, researchers have introduced many novel time series visualizations. Wu et al. introduced formation flow to visualize the dynamics of formation transitions in football. Chen et al. designed GameFlow to display game scores and team information in basketball games. Wongsuphasawat and Gotz proposed Outflow to display events and results in a game. Since biomechanical data are time series data, we use time series visualizations, such as line graphs, to visualize the temporal features in biomechanical fingerprints.

[0006] Through these current works, we can find that the sports data visual analysis system based on various visualization technologies is relatively mature, but there are still many problems. For example, the acquisition of game data mostly comes from manual annotation of video recordings of public events, which greatly consumes research efforts and limits the data source driven by big data. From video processing, model training to data automatic analysis, it requires a lot of time, hardware resources and human investment in checking data. Due to the characteristics of competitive sports data, different ball sports or track and field sports, different competition systems and even athletes of different levels all require specific model processing, which brings relatively large problems to the promotion and use of the system. In addition, in the process of visual analysis of competitive sports, there is a significant problem that visualization developers are not familiar with sports professional knowledge, while sports professional practitioners (such as coaches) are not proficient in the use of visual analysis systems, and there are barriers to cross-border knowledge communication. How to design efficient and easy-to-understand visualization forms and interactive methods has become one of the challenges that researchers in the field continue to follow up.

[0007] From the perspective of machine learning applications in sports biomechanics research, the biomechanical features extracted by various existing learning-based methods only perform well in specific tasks. This is often not enough for analysts, and how to combine multiple characteristics to comprehensively and comprehensively display the biomechanical characteristics of an athlete becomes a challenge. Summary of the invention

[0008] In view of the above, the purpose of the present invention is to provide a table tennis biomechanical fingerprint visual analysis system based on an attention mechanism to achieve visual analysis of table tennis biomechanical fingerprints.

[0009] To achieve the above-mentioned purpose of the invention, the embodiment provides a table tennis biomechanical fingerprint visual analysis system based on attention mechanism, including a data reorganization module, a data generation module, a data fusion module and a fingerprint visualization module;

[0010] The data reorganization module is used to convert the service data in table tennis into three forms: spatial form, motion form and time form;

[0011] The feature generation module is used to generate overall features, spatial features, kinematic features and temporal features according to the data after form transformation using a model built based on an attention mechanism;

[0012] The feature fusion module is used to integrate all the features generated by the feature generation module into a biomechanical fingerprint for technical identification;

[0013] The fingerprint visualization module is used to visualize the biomechanical fingerprint and support visual interactive analysis.

[0014] Preferably, in the data reorganization module, the three-dimensional tensor s corresponding to the primary service data is k (2δ×5×12) is converted into space form, motion form and time form, corresponding to three two-dimensional tensors with different data shapes. In the space form, the tensor t l (5×12) connected in series to s k In the kinematic form, the shape of this form is 5×24δ; in the kinematic form, first in s k Transposition l , after transposition, l The shape is 12×5, and then the transposed t l In series, the shape of this form is 12×10δ; in time form, at t l The connection tensor d m (1×12), the shape of this form is 2δ×60, where δ is the number of sampling points.

[0015] Preferably, the model constructed based on the attention mechanism includes an input layer, an embedding layer, a BiLSTM network, an attention layer, a fusion layer, an attention layer, a dense layer, and an output layer, wherein the three forms of data pass through the input layer, the embedding layer, the BiLSTM network, and the attention layer to generate attention vectors of the three forms of data as spatial features, kinematic features, and temporal features, respectively. These attention vectors are fused in the fusion layer and sent to an additional attention layer F to generate an overall attention vector as an overall feature. The dense layer performs classification prediction based on the overall attention vector and outputs the prediction result through the output layer.

[0016] Preferably, multiple models are trained based on multiple tasks according to the model built based on the attention mechanism, and the tasks include identifying serving technique, horizontal ball position, vertical ball position, ball rotation type, gender, player, grip method and professional level.

[0017] Preferably, in the feature fusion module, the features obtained by averaging and fusing the overall features, spatial features, kinematic features and temporal features corresponding to each group of service data are used as biomechanical fingerprints, and the confidence interval of each value in the feature vector after average fusion is also calculated.

[0018] Preferably, in the fingerprint visualization module, the visualization interface provided includes a fingerprint view, a review view, and a detailed view, wherein the fingerprint view displays an overview of the fingerprints of each player, the review view displays detailed feature information of the player selected in the fingerprint overview, and the detailed view displays the original biomechanical data corresponding to the detailed feature information selected in the review view.

[0019] Preferably, the fingerprint view includes a fingerprint card and a filter panel, the fingerprint card and the filter panel form a fingerprint overview, and the biomechanical fingerprints of all players are visualized through icons corresponding to the fingerprint cards, wherein the icon corresponding to the fingerprint card is a ring, the ring is divided into three equal parts, and different colors are used to encode spatial features, motion features and time features respectively, each arc is divided into multiple arcs, and each arc encodes a value in an attention vector through a heat map;

[0020] To make it easier to distinguish different features, the color hue of each heat map remains unchanged, and saturation is used to encode the value;

[0021] To better distinguish, the dimensions within the attention vector are grouped by using blanks;

[0022] Icons are placed on the four corners of the fingerprint card to encode the player's profile information. In the upper left corner, a palm-shaped icon is used to encode the player's handedness; in the upper right corner, the direction of the racket icon is used to encode the player's racket type; in the lower left corner, a gender icon is used to encode the player's gender; in the lower right corner, the abbreviation of the professional level is used to encode the player's professional level;

[0023] The filter panel includes multiple rows of filters. The first row of filters supports filtering operations based on model tasks. By choosing to display the attention vector of the model for a specific task, this filtering operation can change the number of tangent arcs; the second row of filters supports filtering operations based on attention dimensions. By choosing to display the dimensions within each attention vector, this filtering operation can change the number of radial arcs.

[0024] Preferably, the review view includes four parts: overview, spatial features, kinematic features, and time features; wherein, for the overview part, the overall feature is a three-dimensional vector, and a pie chart is used to display the weights of the three features, namely, spatial features, kinematic features, and time features, and the center of the pie chart is the model task; for the spatial feature part, a bar chart with error bars is used to display the spatial features, the line with different colors is the average value, and the rectangle represents the standard confidence interval; for the kinematic feature part, the three axes of each kinematic indicator are merged into a group for better distinction; for the time feature part, a line chart with confidence bands is used to display the weight distribution of different time periods.

[0025] Preferably, the detailed view displays the original biomechanical data corresponding to the detailed feature information selected in the review view according to a time series.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The table tennis biomechanical fingerprint visual analysis system provided in the embodiment takes the collected biomechanical data as input and generates the spatial features, motion features and time features of the fingerprint through a learning-based method. The method is based on the attention mechanism and the BiLSTM network to improve the interpretability and performance of the model. At the same time, new icons are designed and an interactive system is developed to support the exploration of the generated fingerprint. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 is a structural schematic diagram of a table tennis biomechanical fingerprint visual analysis system based on an attention mechanism provided in an embodiment;

[0030] Figure 2 is a schematic diagram of a data shape obtained by reorganizing the data provided in the embodiment;

[0031] Figure 3 is a schematic diagram of a model structure used for feature generation provided in an embodiment;

[0032] FIG. 4( a ) and FIG. 4( b ) are schematic diagrams of a visual analysis interaction interface provided by an embodiment;

[0033] Figure 5 is an exploded view of a new icon corresponding to a fingerprint card provided in an embodiment;

[0034] Figure 6 is a diagram showing the specific analysis results of Case 1 provided in the embodiment;

[0035] Figure 7 It is a diagram presenting the specific analysis results of Case 2 provided in the embodiment. DETAILED DESCRIPTION

[0036] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0037] Table tennis is a racket sport, and the player who serves has an advantage over the opponent because the player who serves can control the rhythm of the rebound. Players tend to prioritize improving their serving ability over other techniques. Serving contains more representative features than other techniques. Inspired by this situation, the present invention will generate a player's fingerprint based on the biomechanical data collected when the player serves.

[0038] During the fingerprint research process, it was found that existing fingerprint research can extract highly representative features from biomechanical data because they have shown high performance in various tasks. However, these studies use different models for different tasks. If the present invention focuses on one of the models, then these features are not comprehensive enough. If the present invention includes all models, it is difficult to fuse all features due to the heterogeneity of feature vectors, and how to extract comprehensive features to generate fingerprints is a challenge. To solve this problem, the present invention constructs a new model based on a bidirectional LSTM network (BiLSTM). This new model is used in feature generation to obtain comprehensive features.

[0039] During the fingerprint research, it was also found that the most advanced machine learning technologies such as LSTM are inherently nonlinear, so it is difficult to explain the mechanisms and results of these models. Fingerprints generated based on this technology are naturally confusing. In addition, it is inherently difficult for coaches and players to understand machine learning models. They will doubt the reliability of the generated fingerprints. How to improve the interpretability and credibility of the model is also a challenge. To solve this problem, the present invention applies the attention mechanism to the model to enhance the interpretability of the generated features.

[0040] In the fingerprint research process, the fingerprints generated by machine learning technology are abstract. They are usually represented by high-dimensional vectors or tensors. Although the digital representation can accurately capture the characteristics of the players, it is difficult to analyze such data. How to effectively visualize these fingerprints to support effective exploration is a challenge. To solve this problem, the present invention designs a novel visualization for biomechanical fingerprints and supports interactive visualization to enable exploration and research of fingerprints.

[0041] Based on the research, experts provide a set of requirements that a fingerprint needs to meet from a biomechanical perspective, and summarize the following design considerations based on the requirements:

[0042] (1) Temporal features (C1): Temporal features present the rhythm style of the players. For example, some players have a clean serve action, which leads to a fast serve rhythm. On the contrary, some players' serve action may last for a long time because they use some deceptive actions to confuse their opponents. The batting speed style will affect the opponent's strategy when receiving the serve, which is very important in the game. Therefore, the fingerprint should contain temporal features;

[0043] (2) Spatial features (C2): Spatial features present a player’s body coordination style. Each player has a unique body coordination style. For example, some players’ right wrist movements change most significantly when they serve the ball to different positions. In addition, some players’ left wrist movements also change when they serve with different techniques. Body coordination style reflects each player’s key body joints, which is important information during training. Therefore, the fingerprint should contain the characteristics of body joints;

[0044] (3) Kinematic characteristics (C3): Kinematic characteristics present a player's power style. This style can be described by kinematic indicators such as acceleration, angular velocity, etc. For example, some players have strong hitting power, which can be reflected by larger acceleration measurements. In addition, some players have strong explosive power, which can be reflected by greater changes in angular velocity and acceleration. Power style is also an important information in training. Therefore, the fingerprint should be combined with the characteristics of kinematic indicators;

[0045] (4) Overall characteristics (C4): It is also critical to summarize the overall characteristics of different types of features. This type of characteristics can help experts effectively identify the most important information of a player's fingerprint. Although the fingerprint should provide as many types of features as possible, it should also enable experts to quickly grasp the most important characteristics of a player;

[0046] (5) Comprehensiveness of functions (C5): Since our method is a learning-based method, the features extracted by the model will change as the model task changes. Specifically, when the model is used to identify serving techniques and serving positions, the extracted features will definitely be different. In addition, the features extracted based on different serving data will also be different. Therefore, when generating fingerprints, the features in different situations should be fully considered;

[0047] (6) Feature visualization (C6): All features are usually presented in the form of vectors in the model, which is difficult for experts to understand. Especially when a fingerprint contains various types of features, experts may be overwhelmed when analyzing. Therefore, effective visualization methods should be developed to facilitate the analysis of fingerprints.

[0048] Based on the above invention concepts and requirements, the embodiment provides a table tennis biomechanical fingerprint visual analysis system based on the attention mechanism, such as Figure 1 As shown, it includes a data reorganization module A, a data generation module B, a data fusion module C and a fingerprint visualization module D, wherein the data reorganization module A converts the service data into three forms, namely, spatial form, motion form and time form; the feature generation module B generates overall features, spatial features, kinematic features and time features based on the converted data; the feature fusion module C integrates all features into a fingerprint for technical identification; the fingerprint visualization module D visualizes the fingerprint for further analysis.

[0049] In the embodiment, for the data reorganization module, a data reorganization component is designed according to the spatial characteristics, kinematic characteristics and time characteristics, and the three-dimensional tensor s corresponding to the serve data is reorganized based on the data reorganization component. k (2δ×5×12) is converted into space form, motion form and time form, corresponding to three two-dimensional tensors with different data shapes. Specifically, in the space form, the tensor t l (5×12) connected in series to s k In the kinematic form, the shape of this form is 5×24δ; in the kinematic form, first in s k Transposition l , after transposition, l The shape is 12×5, and then the transposed t l In series, the shape of this form is 12×10δ; in time form, at t l The connection tensor d m (1×12), the shape of this form is 2δ×60, and the shape of the third data form is as follows Figure 2 shown.

[0050] The sensor sampling frequency is 100 Hz, which means there are 100 data sampling points per second. δ sampling points are taken before and after the peak. δ is adjustable. In this implementation, we take δ = 75. k The data size is 2δ*5*12, that is, 150*5*12, 150 represents 150 sampling points, which is in the form of time, 5 represents five sensor positions, which is in the form of space, and 12 represents 12 physical quantities that the sensor can collect, which is in the form of kinematics.

[0051] S is the abbreviation of serve, which means one serve. The data format of one serve is 150*5*12 as mentioned above. This is a tensor, and the order before and after the multiplication sign is meaningful. l It is a tensor whose format (size) is 5*12. The reason why 5 is put in front is because the length of the spatial dimension is 5, and 5 should be reserved. The final format in spatial form is 5*24δ, which is equivalent to extracting the spatial dimension of S and merging the time and kinematic dimensions.

[0052] In the kinematic form, 12 must be extracted, so it is 12*5 (that is, t l The final kinematic format is 12*10δ. l The tensor format is 5×12, so it can be split into 5 vectors, each vector, that is, d m The data format is 1×12.

[0053] exist Figure 2 In the equation, δ is the width of δ from the peak to the back, that is, there are 2δ sampling points in total. Figure 2 The pj in the figure is the peak, and the width to the left and right is δ. The light gray is the abandoned sampling point, and the dark gray is the valid sampling point within the interval of δ on the left and right of the peak. The cross-sectional area of ​​this rectangular cylinder is 5*12, and the length is 2δ, so its volume is 2δ*60, which is the data format in time form. The serving data format directly obtained from the sensor, that is, Sk, is 2δ*5*12.

[0054] In the embodiment, for the data generation module, based on the data in three forms of spatial form, motion form and time form, the purpose of feature generation is to generate three features of spatial feature, kinematic feature and time feature. Due to the powerful performance of recursive neural network in sequence data, the present invention refers to BiLSTM to improve model performance. Compared with ordinary LSTM, BiLSTM captures the background of the past and the future at the same time, which leads to better performance. In addition, the hidden state inside BiLSTM is difficult for experts to understand. In order to solve this problem, the present invention adds an attention mechanism to BiLSTM because it has been used for model interpretation in quite a few cases.

[0055] Specifically, the model structure adopted by the data generation module is as follows: Figure 3 As shown, it includes input layer A, embedding layer B, BiLSTM network C, attention layer D, fusion layer E, attention layer F, dense layer G, and output layer H. The three forms of data pass through the input layer A, embedding layer B, BiLSTM network C, and attention layer D to generate attention vectors of the three forms of data. These attention vectors are fused in the fusion layer E and sent to the additional attention layer F to generate an overall attention vector. The dense layer G performs classification prediction based on the overall attention vector and outputs the prediction result through the output layer H.

[0056] Specifically, the first three layers of the model are demonstrated using the kinematic form as an example. In this process, the overall attention information is obtained.

[0057] We take the kinematic form as an example ( Figure 3 A) is used to show the first three layers. First, the kinematic form data is split into 12 vectors, each with a length of 10δ; then, these vectors are fed into the embedding layer B through the input layer A. The embedding layer B embeds each vector with a fixed hidden size; next, the embedded vectors are fed into the BiLSTM network C in sequence, and the last hidden state of the BiLSTM (the square with a black border) is used to generate the attention vector of the attention layer D. With the attention vector, the context vector is further calculated. After the first four layers, three context vectors with the same length of the data form (hidden size) are generated. In the fusion layer E, the context vectors are combined into a two-dimensional tensor (hiddensize is 3) and fed into an additional attention layer F) to obtain the overall attention. The context vector based on the overall attention is converted into a prediction result vector by the dense layer G. The length of the prediction result vector is the number of classes in the task and is output through the output layer H.

[0058] In this way, spatial features, motion features, temporal features and overall features can be obtained based on the data in spatial form, motion form and temporal form, that is, Figure 3Attention vectors in the layers highlighted in blue, cyan, orange, and black. To ensure comprehensiveness (C5), this model can be trained for different attribute recognition tasks. In this work, 8 models were trained for 8 tasks, including recognizing serve technique, horizontal ball position, vertical ball position, type of ball spin, gender, player, grip, and level of expertise. These tasks were provided by experts, and according to the experts, the recognized attributes were the most important information for identifying the biomechanical characteristics of the players.

[0059] A comparative experiment is also designed to verify the effectiveness of the model design in feature generation.

[0060] Table 1 Mean and standard error of the accuracy (10-fold) of different models during the experiment

[0061]

[0062] The accuracy of the model in feature generation will affect the accuracy of the extracted features. The model of the present invention (BiLSTM-T) was compared with 6 alternatives. Wang et al. mentioned six advanced classification models (i.e., LSTM, Random Forest, Deep Forest, XGBoost, LightGBM) for technology identification. Random Forest, LSTM and LightGBM were added to the candidate list of the present invention. Deep Forest was deleted because it is based on bootstrap aggregation, similar to Random Forest. In addition, in previous studies, Random Forest performed better than it. For similar reasons, the present invention chose LightGBM instead of XGBoost. Since the model of the present invention is based on BiLSTM, BiLSTM is also added to the candidate list. In addition to these four alternatives, the model of the present invention has two other alternatives, BiLSTM-S and BiLSTM-K. The structures of these two models are the same as the model of the present invention. The only difference is the method used to generate Figure 3 The key of the overall attention vector in . The model of the present invention uses the context vector in time form, BiLSTM-S uses the vector in space form, and BiLSTM-K uses the vector in motion form.

[0063] When processing the serving data, δ is set to 75. 70% of the serving data is used as training data, 15% of the data is used as validation data, and the remaining 15% of the data is used as test data. 10-fold cross validation is used to evaluate the performance of each model. The mean and standard error of the accuracy are shown in Table 1. According to the results, the model of the present invention performs better than other models in identifying horizontal ball position, vertical ball position, ball rotation type, grip method, and gender. Especially when identifying horizontal ball position, our model performs much better than other models. When identifying serving techniques, players, and professional levels, the model of the present invention does not show any advantage over the basic BiLSTM. However, the performance difference between the two models is very small and almost the same. Therefore, in summary, the overall performance of the model of the present invention is still relatively outstanding. It can handle various recognition tasks.

[0064] In the embodiment, for the data fusion module, the purpose of feature fusion is to combine various features to produce a comprehensive biomechanical fingerprint (C5). The attention vector of a serve is biased. However, retaining the vectors of all serves would overwhelm the expert. Therefore, for each model, the average attention vector of the four features is calculated based on the corresponding attention vectors of all serves. In this way, 32 (4 features 8 models) average attention vectors are used to construct a player's biomechanical fingerprint; in addition, the confidence interval of each value in the average attention vector is also calculated.

[0065] In the embodiment, for the fingerprint visualization module, an efficient visualization means (C6) is provided for experts to analyze fingerprints. A customized icon is specifically designed to encode the biomechanical fingerprint. The icon can reveal the similarities and differences between multiple fingerprints. Based on this visualization, a visual analysis interaction method is further developed to help experts discover potential patterns in fingerprints.

[0066] As shown in Figure 4(a) and Figure 4(b), the corresponding interfaces of the visual analysis interaction method include fingerprint view A, review view D, and detailed view H, where fingerprint view A displays the fingerprint overview B, fingerprint overview C, etc. of each player, review view D displays the detailed feature information F of the player selected in the fingerprint overview C, and detailed view H displays the original biomechanical data corresponding to the detailed feature information selected in the review view D.

[0067] Specifically, the fingerprint overview in the fingerprint view A visualizes the fingerprints of all players with guideable icons, as shown in B and C in Figure 4(a). Analysts can adjust the content of each icon as needed by using various filters and sliders. After selecting the fingerprint of interest, the review view D will display detailed attention information of the four features, as shown in Figure 4(b). E and G in Figure 4(b) are spatial features of other athletes, and analysts can examine the feature differences between different players or tasks. In addition, analysts can further select specific dimensions in each feature and examine the raw biomechanical data in the detailed view G. The system is a web-based application, specifically using React.js to develop the front end and the Django framework to develop the back end that generates all fingerprints.

[0068] In the embodiment, the fingerprint view A in FIG4(a) includes a fingerprint card and a filter panel. The fingerprint card shown in FIG4(a) B expresses the player's fingerprint and his / her personal information. The new icon corresponding to the fingerprint card is designed to encode the player's fingerprint. Figure 5 Shown is a new icon, where A is a radar chart that encodes the overall features, B, D, and E use segmented arcs with heat maps to encode spatial features, motion features, and temporal features, respectively, and G, H, and I are fingerprints under different filtering conditions.

[0069] In such Figure 5 In the new icon shown, a ring is divided into three equal parts (120 ° ), using different colors (e.g., blue, cyan, and orange) to encode spatial features, motion features, and temporal features. Since each feature contains multiple (e.g., eight) attention vectors, each arc is divided into multiple (e.g., eight) arcs, and each arc encodes the value of an attention vector through a heat map, such as Figure 5 B, D, E. To distinguish different features, the color hue of each heat map remains unchanged, and saturation is used to encode the value. To better distinguish, the dimensions in the attention vector are grouped by using blanks, such as Figure 5 As shown in C.

[0070] like Figure 5 As shown in B, the arc presents the attention vector of the skill recognition model within the spatial feature. Most of the attention is on the racket, while the left arm receives the least attention. In the center of the ring, a 32-dimensional radar chart is placed to show the overall features, such as Figure 5 As shown in A, the interior of the radar chart is filled with black to highlight the differences between the overall characteristics of different players. On this basis, two visual channels, the position channel and the direction channel, are also used to distinguish the different characteristics of different models in the icon.

[0071] Through the new icons, as shown in Figure 4(a), icons are placed on the four corners of the fingerprint card to encode the player's profile information. In the upper left corner, a palm-shaped icon is used to encode the player's handedness (i.e., right palm: right-handed, left palm: left-handed). In the upper right corner, the placement direction of the racket icon is used to encode the player's racket type (e.g., horizontal racket: horizontal racket, tilted racket: penhold racket). In the lower left corner, the gender icon is used to encode the player's gender. In the lower right corner, the abbreviation of the professional level is used to encode the player's professional level (i.e., NL: athlete, L1: first-level athlete, L2: second-level athlete). For example, Figure 4(a) B presents the fingerprint card of a female right-handed player who uses a horizontal racket and is a first-level athlete.

[0072] The interactive analyst can click on a group of arcs to select all corresponding features for further analysis. For example, if the analyst selects Figure 5 The arc in B, then Figure 5 The arcs in D and E will also be selected and highlighted. A player's selected task will be displayed in the review view D shown in Figure 4(b).

[0073] The fingerprint view includes a filter panel located at the top of the fingerprint view, including multiple rows of filters, which enable analysts to adjust the fingerprint icon from two perspectives: model task and attention dimension. The first row of filters supports filtering operations based on model tasks. Analysts can choose to display the attention vectors of the model for a specific task. This operation can change the number of tangent arcs, such as Figure 5 The second row of filters supports filtering operations based on attention dimensions. The analyst can choose to display the dimensions within each attention vector. This operation can change the number of radial arcs, such as Figure 5 As shown in H, Figure 5 Figure I presents a fingerprint filtered by two perspective filters. Interactions All filters except the slider support multi-selection. Analysts can select tasks and dimensions as needed. For dimension filtering of time features, analysts can use the slider to select the time period to be displayed in the fingerprint.

[0074] In the embodiment, the review view D in FIG. 4( b) shows the details of the features selected in the fingerprint view A. For example, FIG. 4( b) F shows the features selected in FIG. 4( a) C. Each feature has a separate visualization method for presentation. Specifically, the review view includes four parts: overview, spatial features, kinematic features, and temporal features. Among them, for the overview part, the overall feature is a three-dimensional vector, and the embodiment uses a pie chart to show the weights of the three features of spatial features, kinematic features, and temporal features, and the center of the pie chart is the model task. For example, the pie chart shown in FIG. 4( b) F shows the overall features generated by the technique recognition model based on the serve of player 2, which indicates that when identifying the serve technique of player 2, the model pays the most attention to the temporal features. For the spatial feature part, a bar chart with error bars is used to show the spatial features, the dark blue line is the average value, and the rectangle represents the standard confidence interval. For example, as shown in FIG. 4( b) F, the spatial features indicate that the model pays the most attention to the racket of player 2. For the kinematic features, a similar visualization method is used to present the kinematic features, merging the three axes of each kinematic indicator into a group for better distinction. For example, as shown in F in Figure 4(b), the kinematic features show that the model pays the most attention to the magnetic field data of player 2, especially the z-axis. For the temporal features, a line chart with confidence bands is used to show the weight distribution of different time periods. For example, as shown in F in Figure 4(b), the two peaks indicate that most of the model's attention is focused on the middle time period. Interactive analysts can select two dimensions in the spatial graph or the kinematic graph to compare the raw data of the corresponding dimensions in the detailed view. They can click on the corresponding bar in the chart, and the selected dimension will be highlighted. For example, as shown in E in Figure 4(b) and G in Figure 4(b), the right wrist in the spatial graph of player 1 and player 2 is selected.

[0075] In an embodiment, the detail view H in FIG4(b) mainly displays the biomechanical data of the serve based on a time series. For example, FIG4(b) H shows the biomechanical data of the right wrists of Player 1 and Player 4 selected by E and G in the review view. Since a player has many serves, the data of each serve is not listed separately. The serve data with the same technique and position are aggregated by calculating the average. The analyst can change the technique and position by using the two buttons at the top of the detail view. By checking the biomechanical data, the analyst can verify the findings in the review view.

[0076] In addition, a case study was conducted to evaluate the usability of the system. After the study, the analysts were interviewed to get their feedback on the system. In addition, the players analyzed in the system were interviewed to verify the insights from the study. A case study was specifically conducted on two analysts who served the Chinese National Table Tennis Team for more than three years. The data for the case study are the fingerprints of 12 players employed at the time of data collection. To protect their privacy, we replaced the names of the players with players 1-12. δ was set to 75 to generate the fingerprints.

[0077] First, the fingerprints of all 12 players were checked. Based on the radar graph of each fingerprint, it was found that the fingerprints of players 2 and 5 were almost completely concentrated on spatial features and motion features, such as Figure 6 A and B, which shows that the body coordination style (C2) and power style (C3) of these two players are the best representatives of their biomechanical characteristics. However, their hitting styles do not have any unique characteristics because almost no one pays attention to their temporal characteristics. Therefore, it was decided to focus on one of the two players, namely Player 2. It was found that in terms of spatial characteristics of Player 2, in all model tasks, the model's attention was basically focused on the left and right wrists of Player 2, such as Figure 6 This aroused the curiosity of the analysts. They wanted to know whether this pattern only appeared in the fingerprint of player 2. Therefore, by using a filter, all spatial features of all dimensions except the two wrists were filtered out. It was found that in addition to player 2, player 8 also had the same pattern, as shown in Figure 6 As shown in C in the middle, the analyst checked the profile information of these two players and found that they both used penhold grips, which was different from most other players. The analyst explained that this pattern was caused by different playing styles with different grip types. Compared with players using horizontal rackets, players using penhold grips had more right wrist movements when serving with different techniques, ball positions and spin types.

[0078] However, upon further inspection of the fingerprints of the two players, it was discovered that the model’s focus was not entirely on the right wrist. When identifying serving technique, horizontal ball position, and professionalism, the focus was on the left wrist, as shown in Figure 2. Figure 6 As shown in B. Analysts say that in general, when a player changes the serving technique and ball position, the difference in his / her movements should be concentrated in the right wrist. The difference in the left wrist movement should be relatively small. This is because players need to avoid unnecessary movements to confuse opponents. However, the movements of the left wrists of these two players caught the attention of the model. Analysts believe that the movements of these two players when serving are not perfect. There are too many redundant movements in the left wrist, and they need to reduce these movements in training.

[0079] The embodiment further selects the spatial features generated by the technology recognition model and the horizontal ball position recognition model. Figure 6 The two donuts shown in D and E in the figure. These two models do pay more attention to spatial and kinematic features. In terms of spatial features, the attention paid to the left wrist is significantly higher than that paid to other body joints, including Figure 6 The rackets shown in F and G. Among the kinematic features, the attention paid to the magnetic field is significantly higher than that paid to other kinematic indicators, such as Figure 6 H and I. The analysts interpreted the changes in the magnetic field as reflecting changes in the players’ range of motion. The high focus indicated that Player 2 experienced significant changes in his range of motion when using different serving techniques, and these changes were reflected in the left wrist. To verify this conclusion, the Y-axis of the magnetic field was selected in both models to examine the raw biomechanical data of different body joints along the Y-axis of the magnetic field in the detailed view. After examining the data for different serving techniques and horizontal ball positions, the analysts’ inferences were confirmed.

[0080] After investigating spatial and kinematic features, it was decided to analyze the temporal features in the fingerprints. The temporal features of all players were checked and it was found that the temporal features of the players were quite diverse. Therefore, the analysts suggested filtering out the temporal features by task. Tasks not relevant to technique recognition were deselected in the filter. After filtering, it was found that in the temporal features of most players, the model's attention was mainly focused on the middle of the serving process, such as Figure 7 This suggests that the biomechanical data from the period when the racket contacts the ball is key to the model’s ability to differentiate between different skill and expertise levels. Analysts say players with this pattern must be high-level players because they don’t make unnecessary movements at any time other than when hitting the ball.

[0081] Some exceptions were found. Figure 7 The fingerprint of player 4 shown in B shows that in the task of identifying the type of rotation, the model's attention is mostly focused on the beginning of the serve process (the inner arc is darker), which is the stage when the player throws the ball. This temporal feature is selected, and the peak at the beginning of the serve is found in the review view, such as Figure 7 As shown in Figure E. The analyst explained that when serving with different types of spin, the player's movements change mainly before the ball is hit by the racket. The movements at this stage often have a subtle effect on the spin because the spin is mainly determined by the contact between the racket and the ball. Therefore, it can be inferred that the spin quality of the player's serve is not high enough. In addition, the difference in movements at this stage can also provide clues to the opponent, allowing them to predict the type of spin of the incoming ball in advance. This is a disadvantage for players in the game. In the spatial features, the model focuses on the left wrist, such as Figure 7 As shown in D.

[0082] In addition, we found several examples where attention was focused on the second half of the serving process. For example, when the model identified the spin type of player 6, attention became more focused on the last stage of the serving process, such as Figure 7 As shown in C. After selecting this time point, the line graph in the review view does show a continuous upward trend, as shown in Figure 7 As shown in G in the figure. The analyst explained that after the ball is hit by the player with the racket, the player's follow-up action will change according to the type of rotation of the ball. Such a situation is also not conducive to high-quality serves. Player 6 should improve it in subsequent training. It is also found that the error bar of this player's spatial characteristics is quite long, as shown in Figure 7 As shown in Figure F, it shows that the model's attention often changes when identifying the player's rotation type. This indirectly shows that during each serve, the coordination of the player's body joints is not very stable, and its stability should be strengthened in future training.

[0083] In the embodiment of the present invention, the definition of table tennis biomechanical fingerprint is proposed, and a system is developed to help experts generate and analyze players' fingerprints. The system takes the biomechanical data collected by the IMU device as input and generates the spatial features, motion features and temporal features of the fingerprint through a learning-based method. The method is based on the attention mechanism and BiLSTM network to improve the interpretability and performance of the model. At the same time, new icons are designed, and an interactive system is developed to support the exploration of the generated fingerprint.

[0084] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A table tennis biomechanical fingerprint visual analysis system based on attention mechanism, characterized in that: It includes data reorganization module, feature generation module, feature fusion module and fingerprint visualization module; The data reorganization module is used to convert the service data in table tennis into three forms: spatial form, motion form and time form, including: converting the three-dimensional tensor s corresponding to a serving data into k (2δ×5×12) is converted into space form, motion form and time form, corresponding to three two-dimensional tensors with different data shapes. In the space form, the tensor t l (5×12) concatenated to the three-dimensional tensor s k In the kinematic form, the shape of this form is 5×24δ; in the kinematic form, first in s k Transposition l , after transposition, l The shape is 12×5, and then the transposed t l In series, the shape of this form is 12×10δ; in time form, at t l The connection tensor d m (1×12), the shape of this form is 2δ×60, where δ is the number of sampling points; The feature generation module is used to generate overall features, spatial features, kinematic features and temporal features according to the data after the form conversion using a model built based on the attention mechanism, wherein multiple models are trained based on multiple tasks according to the model built based on the attention mechanism, and the tasks include identifying serving techniques, horizontal ball positions, vertical ball positions, ball rotation types, genders, players, grip methods and professional levels; The feature fusion module is used to integrate all the features generated by the feature generation module into a biomechanical fingerprint for technical identification, including: averaging and fusing the overall features, spatial features, kinematic features and temporal features corresponding to each set of service data as the biomechanical fingerprint, and also calculating the confidence interval of each value in the feature vector after the average fusion; The fingerprint visualization module is used to visualize the biomechanical fingerprint and support visual interactive analysis.

2. The table tennis biomechanical fingerprint visual analysis system based on the attention mechanism according to claim 1 is characterized in that: The model built based on the attention mechanism includes an input layer, an embedding layer, a BiLSTM network, an attention layer, a fusion layer, an attention layer, a dense layer, and an output layer. The three forms of data pass through the input layer, the embedding layer, the BiLSTM network, and the attention layer to generate attention vectors of the three forms of data as spatial features, kinematic features, and temporal features, respectively. These attention vectors are fused in the fusion layer and sent to an additional attention layer F to generate an overall attention vector as an overall feature. The dense layer performs classification prediction based on the overall attention vector and outputs the prediction result through the output layer.

3. The table tennis biomechanical fingerprint visual analysis system based on attention mechanism according to claim 1 is characterized in that: In the fingerprint visualization module, the visualization interface provided includes a fingerprint view, a review view, and a detailed view, wherein the fingerprint view displays an overview of the fingerprints of each player, the review view displays the detailed feature information of the player selected in the fingerprint overview, and the detailed view displays the original biomechanical data corresponding to the detailed feature information selected in the review view.

4. The table tennis biomechanical fingerprint visual analysis system based on the attention mechanism according to claim 3 is characterized in that: The fingerprint view includes a fingerprint card and a filter panel, the fingerprint card and the filter panel form a fingerprint overview, and the biomechanical fingerprints of all players are visualized through the icons corresponding to the fingerprint cards, wherein the icon corresponding to the fingerprint card is a ring, which is divided into three equal parts, and different colors are used to encode spatial features, motion features and time features respectively, each arc is divided into multiple arcs, and each arc encodes a value in an attention vector through a heat map; The color hue of each heat map remains constant, and saturation is used to encode the value; Dimensions within the attention vector are grouped by using whitespace; Icons are placed on the four corners of the fingerprint card to encode the player's profile information. In the upper left corner, a palm-shaped icon is used to encode the player's handedness; in the upper right corner, the direction of the racket icon is used to encode the player's racket type; in the lower left corner, a gender icon is used to encode the player's gender; in the lower right corner, the abbreviation of the professional level is used to encode the player's professional level; The filter panel includes multiple rows of filters. The first row of filters supports filtering operations based on model tasks. By selecting to display the attention vector of the model for a specific task, this filtering operation changes the number of tangent arcs; the second row of filters supports filtering operations based on attention dimensions. By selecting to display the dimensions within each attention vector, this filtering operation changes the number of radial arcs.

5. The table tennis biomechanical fingerprint visual analysis system based on attention mechanism according to claim 3 is characterized in that: The review view includes four parts: overview, spatial features, kinematic features, and time features. Among them, for the overview part, the overall feature is a three-dimensional vector, and a pie chart is used to display the weights of the three features, namely, spatial features, kinematic features, and time features. The center of the pie chart is the model task. For the spatial feature part, a bar chart with error bars is used to display the spatial features, the line with different colors is the average value, and the rectangle represents the standard confidence interval. For the kinematic feature part, the three axes of each kinematic indicator are merged into a group. For the time feature part, a line chart with confidence bands is used to display the weight distribution of different time periods.

6. The table tennis biomechanical fingerprint visual analysis system based on attention mechanism according to claim 3 is characterized in that: The detailed view displays the original biomechanical data corresponding to the detailed feature information selected in the review view according to a time series.

Images