A method and device for recognizing tennis actions based on deep learning

Abstract

A deep learning-based method and apparatus for tennis motion recognition includes: locating the tennis ball strike moment through acoustic event detection; extracting a certain number of frames and inputting them into a motion recognition network; using a convolutional neural network to extract spatial features from each frame of a video sequence of tennis techniques to be detected; using deformable convolution to capture features related to object deformation; enhancing spatial features through channel attention to focus on key features and obtain feature vectors representing player movement information; inputting the spatial feature vectors of each frame into a Transformer to calculate their temporal correlation; using relative position encoding to simulate the degree of deviation between actions in different frames to strengthen the correlation between different frames; and finally using a multilayer perceptron as a classifier to classify the techniques. This invention uses acoustic event detection to locate the strike moment, combines convolutional neural networks and Transformers to process spatial and temporal information separately, and uses a multilayer perceptron as a classifier. This method can efficiently and quickly identify the techniques used by tennis players.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and behavior recognition technology, and relates to a method and device for tennis motion recognition based on deep learning. Background Technology

[0002] Tennis is a globally popular sport with extremely high commercial value. Identifying and statistically analyzing the hitting motions used by players can, to some extent, determine a player's technical style. If computer-based automatic identification and statistical analysis of hitting motions could be achieved, it would save a significant amount of time and manpower, making it easier for coaches to provide guidance to players based on the statistical data.

[0003] In recent years, numerous methods have been applied to tennis motion recognition. Some methods collect data by installing sensors on the racket and combine them with various machine learning techniques to identify the motion; however, installing sensors significantly affects the player's feel for the shot. Other methods combine the player's position relative to the ball to determine the hitting motion, but the tennis ball is too small, often resulting in missed targets. Still other methods identify the motion by calculating the angles between the player's joints in keyframes; this method requires extensive manual design and does not consider the connection between frames before and after a shot. Therefore, designing a more convenient and efficient tennis motion recognition method has become an urgent need. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a tennis motion recognition method based on deep learning.

[0005] This invention uses acoustic event detection to locate the moment a tennis ball is struck. A convolutional neural network is used to extract spatial features from each frame of the input sequence of techniques to be identified. Deformable convolution and channel attention are combined to enhance the spatial features, resulting in a feature vector representing the player's movement information. The spatial feature vector of each frame is input into a Transformer to calculate their temporal correlation. Relative position encoding is used to strengthen the correlation between frames, simulating the relative offset of each frame's technique. Finally, a multilayer perceptron is used as a classifier to classify the techniques, achieving deep learning-based tennis technique recognition.

[0006] This invention is achieved through the following scheme:

[0007] In a first aspect, the present invention provides a tennis motion recognition method based on deep learning, comprising:

[0008] S10: Use acoustic event detection to locate the time of the ball being hit, and extract a certain number of frames of images as the action sequence to be recognized and input them into the action recognition network.

[0009] S20 uses a convolutional neural network to extract spatial features from each frame of the input tennis technique video sequence, and combines deformable convolution and channel attention to enhance the spatial features;

[0010] S30, input the spatial feature vector of each frame into the Transformer to calculate the temporal relationship between vectors, and introduce relative position coding to strengthen the correlation between frames;

[0011] S40 uses a multilayer perceptron for technical action classification.

[0012] Furthermore, S10 specifically includes:

[0013] Acoustic event detection is used to locate the time of the ball strike. A certain number of frames before and after the ball strike are extracted and input into the action recognition network as the sequence of actions to be recognized.

[0014] Specifically, S20 is:

[0015] A convolutional neural network for extracting spatial features was constructed. The main body of the network is MobileNetV3. Deformable convolutional modules were constructed to replace the post-layers in MobileNetV3 to further compute the feature maps, and a channel attention mechanism was used to assign weights to the feature maps of different channels.

[0016] Furthermore, the deformable convolution module used is composed of 1x1 convolution, Deformable Convolution Network (DCN), HardSwish activation function, BatchNorm, and channel attention.

[0017] The deformable convolutional network used learns the offset between each pixel, making the model more sensitive to the overall shape of the object during sampling. The offset is learned through a convolutional layer with the same receptive field as the convolutional kernel.

[0018] The channel attention used performs global pooling and average pooling on the input feature map, respectively. These two vectors are then fed into the fully connected layer to obtain two new vectors. These two vectors are added together and then passed through the sigmoid activation function to obtain the weight values ​​ω for different channels. The feature map is then multiplied by the weight values ​​ω to obtain a new feature map.

[0019] Specifically, S30 is:

[0020] All vectors obtained from S20 are used as input to the Transformer, and an additional vector is added for the final action classification.

[0021] Attention is calculated on all input vectors, and relative position encoding is used during the calculation to represent the relative distance between each vector, in order to simulate the offset of the technical action at different times.

[0022] Furthermore, the relative position encoding used is generated by sine and cosine functions.

[0023] Specifically, S40 is:

[0024] After the calculation in step S30, the vectors used for action recognition are extracted and input into the multilayer perceptron for technical action classification. The action recognition network is then trained using the cross-entropy loss function.

[0025] Secondly, embodiments of this disclosure also provide a tennis motion recognition device based on deep learning, comprising:

[0026] At least one processor;

[0027] Memory that communicates with the processor;

[0028] The memory stores instructions that can be executed by at least one processor, which are executed by at least one processor to enable at least one processor to perform the deep learning-based tennis technique action recognition method in the first aspect described above.

[0029] A third aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the deep learning-based tennis technique motion recognition method of the present invention.

[0030] This invention, through its solution, locates the time of the shot using acoustic event detection, extracts spatial features using a convolutional neural network, calculates the time information between actions using a Transformer, and classifies technical actions using a multilayer perceptron, thereby achieving accurate recognition of tennis technical actions. This can, to a certain extent, solve the problems of complex operation and low robustness of existing tennis action recognition methods.

[0031] The beneficial effects of this invention are: it eliminates the need for manual design and enables convenient and efficient recognition of tennis movements. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the implementation will be briefly described below:

[0033] Figure 1 This is a flowchart illustrating the tennis motion recognition system provided in an embodiment of the present invention.

[0034] Figure 2 This is a diagram of a tennis motion recognition network architecture provided in an embodiment of the present invention.

[0035] Figure 3 A convolutional neural network architecture diagram provided for embodiments of the present invention;

[0036] Figure 4 A schematic diagram of a deformable convolution module provided in an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of deformable convolution provided in an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram of channel attention provided in an embodiment of the present invention;

[0039] Figure 7 This is a schematic diagram of Transformer attention calculation provided in an embodiment of the present invention.

[0040] Figure 8 This is a schematic diagram of the apparatus provided in an embodiment of the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0042] Example 1

[0043] This embodiment provides a deep learning-based method for recognizing tennis techniques. The method uses acoustic event detection to locate the moment of the tennis ball strike, uses a convolutional neural network to extract spatial features from each frame of the input video sequence, then inputs the obtained spatial vector into a Transformer to calculate temporal information, and finally uses a multilayer perceptron to classify the techniques, thereby achieving accurate recognition of tennis techniques.

[0044] Figure 1 This is a flowchart of a deep learning-based tennis motion recognition system provided by the present invention. This system corresponds to the deep learning-based tennis motion recognition method of the present invention. The specific steps of this method are as follows:

[0045] S10: For the input tennis video, the moment when the player hits the ball is located through acoustic event detection, and the start and end times of the sound of the ball hitting are obtained. The acoustic event detection can be implemented by technologies such as Kaldi and Specialized Decision Surface (SDS).

[0046] Taking a video of 30 frames per second as an example, the midpoint between the start time and the end time is taken as the moment of the hit. The frame closest to this time is selected, and 16 frames are taken forward as the preparation process for the hit, and 13 frames are taken backward as the follow-through process of the finishing motion, for a total of 30 frames as the sequence of hit actions to be identified.

[0047] S20: Spatial features are extracted for each frame in a 30-frame video sequence of tennis technique movements;

[0048] like Figure 3 The diagram shows the convolutional neural network used in this invention. This network is based on the Large version of MobileNetV3. In order to further reduce the number of network parameters and improve the sensitivity to features, deformable convolutional modules are constructed to replace the fourth and subsequent 5x5 convolutions used in the baseline model.

[0049] After the image passes through the front layer of the baseline model, the feature map is transformed by the deformable convolution module. Use max pooling to transform the feature map into Convolution operations are performed using deformable convolution modules, followed by 1x1 convolutions and the HardSwish function to transform the feature maps. After channel attention calculation, global average pooling is finally used to transform the feature map into...

[0050]

[0051] like Figure 4 The diagram shows the deformable convolution module used in this invention. The input feature map first undergoes 1x1 convolution, batch normalization, and HardSwish activation function. After deformable convolution operation, it is batch normalized again and then channel attention is performed after HardSwish function operation. If the size of the feature map after calculation is the same as that before calculation, the feature map after calculation is residually connected to the original feature map.

[0052] like Figure 5 The diagram shows the deformable convolution used in this invention. The deformable convolution learns the offset between each pixel using a convolutional layer with the same receptive field as the kernel. After obtaining the offset, the deformable convolution uses these offsets to adjust the sampling position of the kernel and samples from the input feature map using bilinear interpolation. Its computational expression is as follows:

[0053]

[0054] Where Δp k It's the offset, w′ k (p i ) is the learned weight, x(p+p i ) is the input feature map at position p+p i The value of , R is the receptive field of the convolution kernel, w(p k ) is the convolution kernel at position p k The weights, x(p+p) k+Δp k y is the value of the input feature map at the adjusted sampling position. p It is the value of the output feature map at position p.

[0055] like Figure 6 The diagram shows the channel attention mechanism used in this invention. It assigns weights to each channel of the feature map through computation, and its computational expression is as follows:

[0056] F1 = MLP(MAX(F)) (3)

[0057] F2 = MLP(AVG(F)) (4)

[0058] ω=Sigmoid(F1+F2) (5)

[0059] Where MLP represents multilayer perceptron, MAX represents global max pooling, AVG represents global average pooling, and ω is the final weight of each channel.

[0060] The feature map obtained from the above steps is flattened to obtain 30 feature vectors of length 512, which are used as inputs to the time feature calculation layer.

[0061] S30: Initialize a vector of length 512 for the final action classification, take it and the above 30 feature vectors as input, add position encoding and use Transformer to perform temporal feature extraction. The position encoding used here is randomly generated and can be learned through network training.

[0062] In order to strengthen the correlation between adjacent frames and further improve the accuracy of recognition, the relative position encoding is included in the calculation when performing attention calculation.

[0063] like Figure 7 The diagram shows the Transformer attention calculation method combined with relative position encoding used in this invention. The relative position encoding is used as a bias term in the attention calculation.

[0064] The formula for calculating attention is as follows:

[0065]

[0066] Among them W Q W K W V The query matrix, key matrix, and value matrix generated for all vectors, where dh represents the dimension of each attention head. and The relative positions of the i-th feature vector and other vectors are encoded in the same way, z. iThe number of attention heads is set to 16 for the new vector obtained after calculation.

[0067] The relative position encoding used is generated in the following table:

[0068] Table 1

[0069] pos Position of vector i Position of vector j distance 1 i i-4 -4 …… …… …… …… 5 i i 0 …… …… …… …… 9 i i+4 4

[0070] As shown in Table 1, the relative position codes between vector i and other vectors are generated according to this table. If the distance between the two is less than -4, then its pos is also 1; if the distance between the two is greater than 4, then its pos is also 9.

[0071] After obtaining the pos of the current vector i and all other vectors, start calculating the relative position codes between them;

[0072] The relative position code is generated by the sine and cosine functions, and their generators are as follows:

[0073]

[0074] Where R (pos,2a) The value at position 2a represents the relative position code of the number pos, and dh represents the dimension of each attention head.

[0075] S40: Extract the vectors used for action classification and use them as input to the classifier for tennis technique action classification. The classifier used has an input layer length of 512, an intermediate layer length of 256, and an output layer length of 12. The action recognition network is trained using the tennis technique action dataset THETIS to obtain the tennis technique action recognition model, and the cross-entropy loss function is used as the loss function for network training.

[0076] The formula for the cross-entropy loss function used is as follows:

[0077]

[0078] Where N represents the number of training samples, Y represents the number of classification actions, and y ic p represents the probability that sample i belongs to c. ic This represents the probability that the model predicts sample i belongs to c.

[0079] The proposed method can classify 12 refined tennis strokes: two-handed backhand, one-handed backhand, backhand volley, backhand slice, forehand shot, open forehand shot, forehand volley, forehand slice, topspin serve, flat serve, sidespin serve, and smash.

[0080] Example 2

[0081] Reference Figure 8This embodiment provides a tennis motion recognition device based on deep learning, comprising: at least one processor; and a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the tennis motion recognition method based on deep learning as described above. Figure 8 The diagram shown is a hardware structure diagram of a device provided in an embodiment of the present invention, except that... Figure 8 In addition to the processor and memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0082] Example 3

[0083] This embodiment provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the deep learning-based tennis motion recognition method described above. The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart memory card, SD card, flash memory card, etc., equipped on the device. Furthermore, the computer-readable storage medium may include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0084] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A deep learning-based tennis motion recognition method, comprising: S10, using acoustic event detection to locate the moment of the ball being struck, extracting a certain number of frames before and after the moment of the ball strike as the action sequence to be identified and inputting it into the action recognition network; the action recognition network includes a convolutional neural network module, a Transformer module, and a classifier; the classifier is a multilayer perceptron. S20, a convolutional neural network is used to extract spatial features from each frame of the input sequence of actions to be identified, and deformable convolution and channel attention are combined to enhance the spatial features; the convolutional neural network module uses the Large version of MobileNetV3 as the baseline model, and a deformable convolution module is constructed to replace the fourth use of 5X5 convolution and the subsequent parts of the baseline model; the post-layer specifically includes a deformable convolution module, a max pooling layer, a 1X1 convolution, a HardSwish function, a channel attention layer, and a global average pooling layer; The deformable convolution module includes 1x1 convolution, deformable convolution DCN, HardSwish activation function, BatchNorm, and channel attention. S30: Input the spatial feature vector of each frame into the Transformer to calculate the temporal relationship between the vectors and obtain the vector for action recognition; introduce relative position coding to enhance the correlation between frames; S40 inputs the vectors used for action recognition into the multilayer perceptron for technical action classification.

2. The tennis motion recognition method based on deep learning as described in claim 1, characterized in that, Step S10 is as follows: Acoustic event detection is used to locate the time of the ball strike. A certain number of frames before and after the ball strike are extracted and used as the sequence of actions to be identified, which are then input into the action recognition network.

3. The tennis motion recognition method based on deep learning as described in claim 1, characterized in that, Step S20 is as follows: A convolutional neural network for extracting spatial features was constructed. The main body of the network is MobileNetV3. Deformable convolutional modules were constructed to replace the post-layers in MobileNetV3 to further compute the feature maps, and a channel attention mechanism was used to assign weights to the feature maps of different channels.

4. The tennis motion recognition method based on deep learning as described in claim 3, characterized in that, The deformable convolution module used consists of a combination of 1x1 convolution, deformable convolution network (DCN), hardSwish activation function, BatchNorm, and channel attention. The deformable convolutional network used learns the offset between each pixel, making the model more sensitive to the overall shape of the object during sampling. The offset is learned through a convolutional layer with the same receptive field as the convolutional kernel. The channel attention method performs global pooling and average pooling on the input feature map, then inputs these two vectors into a fully connected layer to obtain two new vectors. These two vectors are then summed and passed through a sigmoid activation function to obtain the weights for different channels. The feature map and weight values ​​are combined. Multiplying them yields a new feature map.

5. The tennis motion recognition method based on deep learning as described in claim 1, characterized in that, Step S30 is as follows: All vectors obtained from S20 are used as input to the Transformer, and a multilayer perceptron is added for the final action classification. Attention is calculated on all input vectors, and relative position encoding is used during the calculation to represent the relative distance between each vector, in order to simulate the offset of the technical action at different times.

6. The tennis motion recognition method based on deep learning as described in claim 5, characterized in that, The relative position encoding used is generated by sine and cosine functions.

7. The tennis motion recognition method based on deep learning as described in claim 1, characterized in that, Step S40 is as follows: After the calculation in step S30, the vector used for action recognition is input into the multilayer perceptron for technical action classification, and the action recognition network is trained using the cross-entropy loss function.

8. A tennis motion recognition device based on deep learning, comprising: At least one processor; Memory that communicates with the processor; The memory stores instructions that can be executed by at least one processor, which, when executed by at least one processor, enables the at least one processor to perform the deep learning-based tennis technique motion recognition method according to any one of claims 1-7.

9. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the deep learning-based tennis technique motion recognition method according to any one of claims 1-7.

Images