Tennis action recognition method and device based on deep learning

Abstract

The invention discloses a tennis action recognition method and device based on deep learning, and the method comprises the steps: detecting and positioning a tennis hitting moment through an acoustic event, extracting a certain number of frames, and inputting the frames into an action recognition network; the method comprises the following steps: performing spatial feature extraction on each frame in a to-be-detected tennis skill action video sequence by using a convolutional neural network, capturing features related to object deformation by using deformable convolution, enhancing the spatial features through channel attention, focusing key features, and obtaining feature vectors capable of representing player movement information; and inputting the spatial feature vector of each frame into Transform to calculate the correlation of the spatial feature vector in time, simulating the deviation degree between different frame actions by using relative position coding, enhancing the correlation between different frames, and finally performing technical action classification by using a multi-layer perceptron as a classifier. According to the method, the hitting moment is positioned through acoustic event detection, space information and time information are processed separately in combination with a convolutional neural network and a Transform, and a multi-layer perceptron is used as a classifier. By means of the method, the skilled actions used by the tennis player can be efficiently and rapidly recognized.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and behavior recognition, and relates to a tennis action recognition method and device based on deep learning. Background Art

[0002] Tennis is a sport that is popular all over the world and has extremely high commercial value. Identifying and counting the hitting movements used by players in sports can determine the players' technical style to a certain extent. If the automatic identification and statistics of hitting movements can be achieved through computers, a lot of time for human identification and statistics can be saved, which can make it easier for coaches to guide players based on the statistics.

[0003] In recent years, many methods have been used for tennis action recognition. Some methods collect data by installing sensors on the racket and combine various machine learning methods to recognize actions, but installing sensors will greatly affect the player's feel for hitting the ball. Some methods combine the position of the person and the ball to judge the player's hitting action, but the tennis target is too small and often loses the target. Some methods recognize actions by calculating the angles of human joints in key frames. This method requires a lot of manual design and does not take into account the connection between the frames before and after the player completes a hitting action. Therefore, designing a more convenient and efficient tennis action recognition method has become an urgent need. Summary of the invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a tennis motion recognition method based on deep learning.

[0005] The method of the present invention uses acoustic event detection to locate the moment when a tennis ball is hit, uses a convolutional neural network to extract spatial features of each frame in an input technical action sequence to be identified, combines deformable convolution and channel attention to enhance the spatial features, and obtains a feature vector that can represent the player's motion information; the spatial feature vector of each frame is input into a Transformer to calculate their temporal association, uses relative position encoding to strengthen the association between frames, simulates the relative offset of each frame of technical action, and finally uses a multi-layer perceptron as a classifier to classify technical actions, thereby realizing tennis technical action recognition based on deep learning.

[0006] The present invention is achieved through the following scheme:

[0007] In a first aspect, the present invention provides a tennis action recognition method based on deep learning, comprising:

[0008] S10, using acoustic event detection to locate the time when the ball is hit, extracting a certain number of frames of images as the action sequence to be recognized and inputting them into the action recognition network;

[0009] S20, uses a convolutional neural network to extract spatial features from each frame of the input tennis technical action video sequence, and combines deformable convolution and channel attention to enhance the spatial features;

[0010] S30, inputting the spatial feature vector of each frame into the Transformer to calculate the temporal relationship between the vectors, and introducing relative position coding to strengthen the correlation between frames;

[0011] S40, technical action classification using multi-layer perceptron.

[0012] Furthermore, the S10 is specifically:

[0013] Acoustic event detection is used to locate the time when the ball is hit, and a certain number of frames before and after the moment of hitting the ball are extracted and input into the action recognition network as the sequence of technical actions to be recognized.

[0014] The S20 is specifically:

[0015] A convolutional neural network for extracting spatial features is built. The main body of the network is MobileNetV3. A deformable convolution module is constructed to replace the post-layer in MobileNetV3 to further calculate the feature map, and the channel attention mechanism is used to assign weights to the feature maps of different channels.

[0016] Furthermore, the deformable convolution module used is composed of 1x1 convolution, deformable convolution Network (DCN), HardSwish activation function, BatchNorm and channel attention.

[0017] The deformable convolutional network used learns the offset between each pixel, making the model more sensitive to the overall shape of the object when sampling. The offset is learned through a convolutional layer with the same receptive field as the convolution kernel.

[0018] The channel attention used performs global pooling and average pooling on the input feature map, and then inputs the two vectors into the fully connected layer to obtain two new vectors. After adding the two vectors, the sigmoid activation function is used to obtain the weight value ω of different channels, and the feature map is multiplied by the weight value ω to obtain a new feature map.

[0019] The S30 is specifically:

[0020] All vectors obtained by S20 are used as input to the Transformer, and an additional vector is added for the final action classification.

[0021] Attention calculation is performed on all input vectors, and relative position encoding is used during the calculation to represent the relative distance between each vector to simulate the offset of technical actions at different times.

[0022] Furthermore, the relative position code used is generated by a sine function and a cosine function.

[0023] The S40 is specifically:

[0024] After calculation in step S30, the vector used for action recognition is extracted and input into a multi-layer perceptron for technical action classification, and the action recognition network is trained using a cross entropy loss function.

[0025] In a second aspect, the present disclosure also provides a tennis action recognition device based on deep learning, including:

[0026] at least one processor;

[0027] a memory communicatively coupled to the processor;

[0028] The memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the tennis technical action recognition method based on deep learning in the first aspect;

[0029] A third aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the tennis technical action recognition method based on deep learning of the present invention.

[0030] The present invention uses the scheme of the present invention to locate the hitting time through acoustic event detection, uses a convolutional neural network to extract spatial features, uses a Transformer to calculate the time information between actions, and uses a multi-layer perceptron to classify technical actions, thereby achieving accurate recognition of tennis technical actions, which can to a certain extent solve the problems of complex operation and low robustness of existing tennis action recognition methods.

[0031] The beneficial effect of the example of the present invention is that tennis movements can be recognized conveniently and efficiently without manual design. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solution of the embodiment of the present disclosure, the following is a brief introduction to the drawings required for implementation:

[0033] Figure 1 A schematic diagram of the process flow of a tennis motion recognition system provided by an embodiment of the present invention;

[0034] Figure 2 A diagram of a tennis motion recognition network architecture provided by an embodiment of the present invention;

[0035] Figure 3 A convolutional neural network architecture diagram provided for an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of a deformable convolution module provided by an embodiment of the present invention;

[0037] Figure 5 A schematic diagram of deformable convolution provided by an embodiment of the present invention;

[0038] Figure 6 A schematic diagram of channel attention provided by an embodiment of the present invention;

[0039] Figure 7 A schematic diagram of Transformer attention calculation provided by an embodiment of the present invention.

[0040] Figure 8 A schematic diagram of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0042] Example 1

[0043] This embodiment provides a method for identifying tennis technical movements based on deep learning. The method uses acoustic event detection to locate the moment of tennis hitting, uses a convolutional neural network to extract spatial features from each frame of an input video sequence, and then inputs the resulting spatial vector into a Transformer for temporal information calculation. Finally, a multi-layer perceptron is used to classify technical movements, thereby achieving accurate recognition of tennis technical movements.

[0044] Figure 1 : is a flow chart of a tennis action recognition system based on deep learning provided by the present invention. The system corresponds to a tennis technical action recognition method based on deep learning of the present invention. The specific steps of the method are:

[0045] S10: For the input tennis sports video, the moment when the player hits the ball is located through acoustic event detection, and the start and end time of the hitting sound are obtained. The acoustic event detection can be implemented through technologies such as Kaldi and Specialized Decision Surface (SDS).

[0046] Taking a video with 30 frames per second as an example, the median of the start time and the end time is taken as the moment of the hitting moment, and the frame closest to the time is selected. The 16 frames forward are taken as the preparation process for the hitting, and the 13 frames backward are taken as the follow-up process of the ending action. A total of 30 frames are used as the hitting action sequence to be identified.

[0047] S20: For each frame in the 30-frame tennis technical action video sequence, a convolutional neural network is used to extract spatial features;

[0048] like Figure 3 The figure shows the convolutional neural network used in the present invention. The network uses the Large version of MobileNetV3 as the benchmark model. In order to further reduce the number of network parameters and improve the sensitivity to features, a deformable convolution module is constructed to replace the fourth 5X5 convolution and subsequent parts of the benchmark model;

[0049] After the image passes through the pre-layer of the baseline model, the feature map is transformed into Use max pooling to transform the feature map into Use the deformable convolution module to perform convolution operations, and then use 1X1 convolution and HardSwish function to transform the feature map into After channel attention calculation, global average pooling is finally used to transform the feature map into

[0050]

[0051] like Figure 4 The figure shows the deformable convolution module used in the present invention. The input feature map first undergoes 1x1 convolution, batch normalization, and HardSwish activation function, and then undergoes batch normalization after the deformable convolution operation. After the HardSwish function operation, the channel attention operation is performed; if the size of the feature map after calculation is the same as that before the operation, the feature map after the operation is residually connected with the original feature map.

[0052] like Figure 5 The figure shows the deformable convolution used in the present invention. The deformable convolution learns the offset between each pixel through a convolution layer with the same receptive field as the convolution kernel. After obtaining the offset, the deformable convolution uses these offsets to adjust the sampling position of the convolution kernel and samples from the input feature map through bilinear interpolation. The operation expression is as follows:

[0053]

[0054] Where Δp k is the offset, w′ k (p i ) is the learned weight, x(p+p i ) is the input feature map at position p+p i The value of, R is the receptive field of the convolution kernel, w(p k ) is the convolution kernel at position p k The weight of x(p+p k+Δp k ) is the value of the input feature map at the adjusted sampling position, y p is the value of the output feature map at position p.

[0055] like Figure 6 The channel attention used in the present invention is shown, which can assign weights to each channel of the feature map through calculation, and its calculation expression is as follows:

[0056] F1=MLP(MAX(F)) (3)

[0057] F2=MLP(AVG(F)) (4)

[0058] ω=Sigmoid(F1+F2) (5)

[0059] Among them, MLP stands for multi-layer perceptron, MAX stands for global maximum pooling, AVG stands for global average pooling, and ω is the weight of each channel finally obtained.

[0060] The feature map obtained in the above steps is flattened to obtain 30 feature vectors with a length of 512 as the input of the temporal feature calculation layer.

[0061] S30: Initialize a vector of length 512 for final action classification, take it and the above 30 feature vectors as input, add position encoding and use Transformer to perform temporal feature extraction operation. The position encoding used here is randomly generated and can be learned through network training;

[0062] In order to strengthen the correlation between adjacent frames and further improve the recognition accuracy, the relative position encoding is also included in the calculation when performing attention calculation;

[0063] like Figure 7 The figure shows a schematic diagram of Transformer attention calculation combined with relative position coding used in the present invention, where the relative position coding is used as a bias item in the attention calculation;

[0064] The public notice for attention calculation is as follows:

[0065]

[0066] Where W Q , W K , W V The query matrix, key matrix, and value matrix generated for all vectors, dh represents the dimension of each attention head, and is the relative position encoding of the i-th eigenvector and other vectors. They are generated in the same way. iis the new vector obtained after calculation, and the number of attention heads is set to 16.

[0067] The relative position encoding used to generate the table is as follows:

[0068] Table 1

[0069] POS The position of vector i The position of vector j distance 1 i i-4 -4 …… …… …… …… 5 i i 0 …… …… …… …… 9 i i+4 4

[0070] As shown in Table 1, the relative position encoding between vector i and other vectors is generated according to this table. If the distance between the two is less than -4, then its pos is also 1; if the distance between the two is greater than 4, then its pos is also 9;

[0071] After obtaining the pos of the current vector i and all other vectors, start calculating the relative position encoding between them;

[0072] The relative position encoding is generated by the sine function and the cosine function, and the generating formulas are as follows:

[0073]

[0074] Where R (pos,2a) Represents the value of the relative position numbered pos encoded at 2a, and dh represents the dimension of each attention head.

[0075] S40: Extract the vector used for action classification and use it as the input of the classifier to classify tennis technical actions. The classifier used has an input layer length of 512, an intermediate layer length of 256, and an output layer length of 12. The action recognition network is trained using the tennis technical action dataset THETIS to obtain a tennis technical action recognition model, and the cross entropy loss function is used as the loss function for network training;

[0076] The cross entropy loss function used is shown as follows:

[0077]

[0078] Where N represents the number of training samples, Y represents the number of classified actions, and y ic represents the probability that sample i belongs to c, p ic Represents the probability that the model predicts that sample i belongs to c.

[0079] The proposed method can classify 12 refined tennis actions, namely two-handed backhand, one-handed backhand, backhand volley, backhand slice, forehand shot, opening forehand shot, forehand volley, forehand slice, topspin serve, flat serve, sidespin serve, and smash.

[0080] Example 2

[0081] Reference Figure 8This embodiment provides a tennis technical action recognition device based on deep learning, comprising: at least one processor; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the tennis technical action recognition method based on deep learning as described above. Figure 8 The figure shows a hardware structure diagram of the device provided by the embodiment of the present invention, except Figure 8 In addition to the processor and memory shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware according to the actual function of the device with data processing capability, which will not be described in detail.

[0082] Example 3

[0083] This embodiment provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the tennis action recognition method based on deep learning as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card, an SD card, a flash memory card, etc. equipped on the device. Furthermore, the computer-readable storage medium can also include an internal storage unit of any device with data processing capabilities, and also include an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0084] It should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A tennis motion recognition method based on deep learning, comprising: S10, using acoustic event detection to locate the time when the ball is hit, extracting a certain number of frames of images as the action sequence to be recognized and inputting them into the action recognition network; S20, uses a convolutional neural network to extract spatial features from each frame of the input tennis technical action video sequence, and combines deformable convolution and channel attention to enhance the spatial features; S30, inputting the spatial feature vector of each frame into the Transformer to calculate the temporal relationship between the vectors, and introducing relative position coding to strengthen the correlation between frames; S40, technical action classification using multi-layer perceptron.

2. A tennis action recognition method based on deep learning as claimed in claim 1, characterized in that: Step S10 is specifically as follows: Acoustic event detection is used to locate the time when the ball is hit, and a certain number of frames before and after the moment of hitting the ball are extracted and input into the action recognition network as the sequence of technical actions to be recognized.

3. A tennis action recognition method based on deep learning as claimed in claim 1, characterized in that: Step S20 is specifically as follows: A convolutional neural network for extracting spatial features is built. The main body of the network is MobileNetV3. A deformable convolution module is constructed to replace the post-layer in MobileNetV3 to further calculate the feature map, and the channel attention mechanism is used to assign weights to the feature maps of different channels.

4. A tennis action recognition method based on deep learning as claimed in claim 3, characterized in that: The deformable convolution module used consists of 1x1 convolution, deformable convolution Convolution Network, DCN), HardSwish activation function, BatchNorm and channel attention; The deformable convolutional network used learns the offset between each pixel, making the model more sensitive to the overall shape of the object when sampling. The offset is learned through a convolutional layer with the same receptive field as the convolution kernel. The channel attention used performs global pooling and average pooling on the input feature map, and then inputs the two vectors into the fully connected layer to obtain two new vectors. After adding the two vectors, the sigmoid activation function is used to obtain the weight value ω of different channels, and the feature map is multiplied by the weight value ω to obtain a new feature map.

5. A tennis action recognition method based on deep learning as claimed in claim 1, characterized in that: Step S30 is specifically as follows: All vectors obtained by S20 are used as input to the Transformer, and an additional vector is added for the final action classification; Attention calculation is performed on all input vectors, and relative position encoding is used during the calculation to represent the relative distance between each vector to simulate the offset of technical actions at different times.

6. A tennis action recognition method based on deep learning as claimed in claim 5, characterized in that: The relative position encoding used is generated by sine and cosine functions.

7. A tennis action recognition method based on deep learning as claimed in claim 1, characterized in that: Step S40 is specifically as follows: After calculation in step S30, the vector used for action recognition is extracted and input into a multi-layer perceptron for technical action classification, and the action recognition network is trained using a cross entropy loss function.

8. A tennis motion recognition device based on deep learning, comprising: at least one processor; a memory communicatively coupled to the processor; The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the tennis technical action recognition method based on deep learning as described in one of claims 1-7.

9. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the tennis technical action recognition method based on deep learning of the present invention.

Images