Pick ball athlete action recognition and analysis system based on multi-modal data fusion

Abstract

The invention discloses a Pick ball athlete action recognition and analysis system based on multi-modal data fusion, and the system comprises the steps: collecting the video image data and sensor data of a Pick ball athlete in the motion process, and carrying out the feature extraction of the body posture of the athlete in the video image data through a convolutional neural network; according to the method, a recurrent neural network is combined to perform action classification on a feature sequence, meanwhile, sensor data are fused, accurate recognition of actions of an athlete is achieved, meanwhile, detailed technical parameters of the athlete in the ball hitting action can be obtained by deeply analyzing the recognized actions, accurate guidance is provided for training of the athlete, and the training efficiency is improved. And the technical level and the competitive ability of athletes can be improved.

Description

[Technical field]

[0001] The present invention relates to the field of sports technology analysis, and in particular to a pickleball player motion recognition and analysis system based on multimodal data fusion. [Background Technology]

[0002] As an emerging comprehensive sport, pickleball has complex and diverse technical movements, including a variety of hitting methods, flexible movement steps and coordinated body postures. In the training and competition process of athletes, accurately identifying and analyzing the movements of athletes is of vital importance to improving technical level, optimizing training programs and improving competition results.

[0003] At present, existing motion recognition technology has certain limitations in the field of pickleball. Traditional methods often rely on a single data source and cannot fully utilize the complementary information of multi-source data, resulting in low accuracy of motion recognition. At the same time, there is a lack of targeted analysis of the unique technical movements and tactical strategies of pickleball, which makes it difficult to meet the needs of athletes and coaches in actual training and competition. Therefore, an innovative system is needed to solve these problems.

[0004] With the development of artificial intelligence and computer vision technology, the application of multimodal data fusion in the field of action recognition has received widespread attention. At present, there are some related patents and research. For example, the patent with publication number CN116934926A proposes a recognition method and system based on multimodal data fusion, which obtains 3D face parameters by performing 3D face reconstruction on face images or videos, extracts high-order speech features using a speech feature extraction network, and establishes a lip correlation coefficient network to achieve a comprehensive expression of the virtual image. However, this method mainly focuses on the fusion of face and voice information, and is not very targeted for the recognition of pickleball players' actions, and does not involve the fusion of body movement information with other modal information.

[0005] For another example, the patent with publication number CN115661561A discloses a multimodal fusion small sample dance motion recognition method and device, which uses a multimodal data fusion method based on video preview guidance, and applies the feature learning method of multimodal data to the small sample dance motion recognition task. However, this method is for small sample dance motion recognition, and its applicability to athlete motion recognition in a specific sport such as pickleball is limited, and it differs from the present invention in the acquisition and fusion methods of multimodal data. [Summary of the invention]

[0006] The present invention overcomes the shortcomings of the prior art and provides a pickleball player motion recognition and analysis system based on multimodal data fusion, thereby achieving accurate recognition and analysis of pickleball player motions.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] The pickleball player motion recognition and analysis system based on multimodal data fusion is characterized by:

[0009] a data acquisition module for collecting video image data of the pickleball player during the exercise and sensor data of the player's movement and the pickleball's movement during the exercise;

[0010] The data preprocessing module is connected to the data acquisition module, and preprocesses the video image data and the sensor data to obtain the video image preprocessing data and the sensor preprocessing data;

[0011] The feature extraction module is connected to the data preprocessing module and uses a convolutional neural network to extract the features of the video image preprocessing data to obtain a video image feature vector;

[0012] The multimodal fusion module is connected to the data preprocessing module and the feature extraction module respectively, and is used to fuse the video image feature vector with the sensor preprocessing data to obtain a fused feature vector;

[0013] The action classification module is connected to the multimodal fusion module and uses a recurrent neural network to classify the fused feature vectors and identify the athlete's action type;

[0014] The motion analysis module is connected to the motion classification module, and analyzes the athlete's motion according to the identified motion type to obtain motion analysis data;

[0015] The training module is connected to the feature extraction module and the action classification module respectively, and trains the convolutional neural network and the recurrent neural network respectively through the athlete's motion data;

[0016] The output module is connected to the motion analysis module and outputs athlete motion recognition results, motion analysis reports and training suggestions based on the motion analysis data.

[0017] The pickle ball player motion recognition and analysis system based on multimodal data fusion as described above is characterized in that: the data acquisition module includes a camera, a pickle ball motion sensor installed on the pickle ball to collect pickle ball motion data, and an athlete motion sensor worn on the athlete to collect athlete motion data. The camera collects video image data, and the pickle ball motion sensor and the athlete motion sensor collect sensor data.

[0018] The pickleball player motion recognition and analysis system based on multimodal data fusion as described above is characterized in that: the data preprocessing module performs normalization preprocessing on the video image data to obtain video image preprocessing data; the data preprocessing module performs denoising, filtering and normalization on the sensor data to obtain sensor preprocessing data.

[0019] The pickleball player motion recognition and analysis system based on multimodal data fusion as described above is characterized in that: the convolutional neural network in the feature extraction module includes a plurality of convolutional layers connected in sequence for learning the posture features of various parts of the athlete's body, and the convolutional layers are connected in sequence to a pooling layer for reducing the data dimension and a fully connected layer for outputting the video image feature vector to the multimodal fusion module.

[0020] The pickleball player motion recognition and analysis system based on multimodal data fusion as described above is characterized in that the convolutional layers use convolution kernels of different sizes and the pooling layers use a maximum pooling operation.

[0021] The pickleball player motion recognition and analysis system based on multimodal data fusion as described above is characterized in that the multimodal fusion module uses splicing or weighted summation to fuse the video image feature vector and the sensor preprocessing data.

[0022] The pickleball player motion recognition and analysis system based on multimodal data fusion as described above is characterized in that: the motion analysis module fits the joint motion trajectory by establishing a kinematic model, and calculates the Pearson correlation coefficient between the body posture data and the hitting effect to analyze the correlation; the motion analysis data includes joint motion trajectory data, correlation data between body posture changes and hitting effects, and divides the action into stages and extracts feature data of each stage.

[0023] The pickleball player motion recognition and analysis system based on multimodal data fusion as described above is characterized in that the training module uses stochastic gradient descent and its variant optimization algorithm combined with back propagation algorithm to train the model.

[0024] The pickleball player action recognition and analysis system based on multimodal data fusion as described above is characterized in that the stochastic gradient descent and its variant optimization algorithms include Adagrad algorithm, Adadelta algorithm or Adam algorithm.

[0025] The pickleball player action recognition and analysis system based on multimodal data fusion as described above is characterized in that the loss function adopts a mean square error or a cross entropy loss function.

[0026] The beneficial effects of the present invention are:

[0027] The present invention collects video image data and sensor data of pickleball players during the movement process, uses convolutional neural network (CNN) to extract features of the players' body postures in the video image data, combines recurrent neural network (RNN) to classify the feature sequence, and fuses the sensor data to achieve accurate recognition of the players' movements. Furthermore, by conducting in-depth analysis of the identified movements, it is possible to obtain detailed technical parameters of the players in the hitting movements, including joint movement trajectories, body posture changes, and movement stage characteristics, etc., to provide accurate guidance for the athletes' training and help improve the athletes' technical level and competitive ability. Compared with the prior art, the present invention focuses more on the pickleball sports scene, is designed for the movement characteristics of pickleball players, can more accurately capture and analyze the details of the athletes' movements, has stronger pertinence and practicality, and has significant novelty and creativity in the field of pickleball player movement recognition. [Drawings]

[0028] Figure 1 It is a schematic diagram of the model structure of the present invention. [Specific implementation method]

[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.

[0030] It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the descriptions of "preferred", "sub-preferred", etc. in the present invention are only used for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "preferred" or "sub-preferred" may explicitly or implicitly include at least one such feature.

[0031] like Figure 1 As shown in the figure, the pickleball player motion recognition and analysis system based on multimodal data fusion includes

[0032] The data acquisition module is used to collect video image data of pickleball players during exercise and sensor data of the player's movement and the pickleball movement during exercise. The data acquisition module includes a camera, a pickleball motion sensor installed on the pickleball to collect the pickleball motion data, and an athlete motion sensor worn on the athlete to collect the athlete's motion data. The camera collects video image data, the pickleball motion sensor and the athlete motion sensor collect sensor data, and the pickleball motion sensor is a micro sensor.

[0033] Preferably, the camera arrangement includes evenly arranging multiple high frame rate (120 frames per second or more) and high resolution (1920×1080 or higher) cameras around a standard pickleball playing field. For example, a camera is arranged at each of the four corners of the field and the midpoint of the long side to ensure full coverage of the field without blind spots, so as to collect clear video image data of athletes during competition or training. The camera is connected to the data preprocessing module via a high-speed and stable Wi-Fi network to ensure real-time transmission of video image data.

[0034] Preferably, the sensor installation and configuration includes installing high-precision and high-sensitivity micro sensors on the ball, and equipping the athletes with motion sensors.

[0035] Among them, the micro sensor installed on the ball is preferably a sensor model that is light in weight, small in size and does not affect the ball's motion performance. During the installation process, ensure that the position and direction of the sensor can accurately collect the ball's flight speed, acceleration, rotation speed, rotation direction and other motion data. The data is transmitted to the data preprocessing module via Bluetooth or a radio frequency signal in a specific frequency band. In order to ensure signal stability, a reasonable signal transmission frequency and power are set. For example, Bluetooth transmission uses low-power Bluetooth technology, the signal transmission frequency is 2.4GHz, and the transmission power is adjusted to an appropriate range (such as between 0dBm-10dBm) according to the size of the venue and interference conditions.

[0036] Before athletes wear sports sensors, they must first perform initial calibration to ensure data accuracy. Sensors are worn on key parts of the body, such as wrists, elbows, shoulders, hips, knees, waists, legs, etc., and are fixed with elastic bandages or specially designed fixtures to ensure that the sensors do not loosen or shift during exercise. The sensors collect information such as the athlete's body posture data (such as body tilt angle, degree of twisting, etc.), movement speed, movement direction, and acceleration, and transmit them to the data preprocessing module wirelessly (such as Bluetooth or radio frequency signals), providing a comprehensive and accurate data source for subsequent analysis.

[0037] The data preprocessing module is connected to the data acquisition module, and preprocesses the video image data and the sensor data respectively to obtain the video image preprocessing data and the sensor preprocessing data. The data preprocessing module performs normalization preprocessing on the video image data to obtain the video image preprocessing data; the data preprocessing module performs denoising, filtering and normalization on the sensor data to obtain the sensor preprocessing data.

[0038] Preferably, in the video image data preprocessing, after the data acquisition module transmits the video image data to the data preprocessing module, the format conversion is first performed. A professional image processing software library (such as OpenCV, etc.) is used to convert the original video image format (such as MP4, AVI, etc.) into a specific image array format suitable for subsequent processing, for example, the image is converted into a three-dimensional array form, where the first dimension represents the number of image frames, the second dimension and the third dimension represent the pixel height and width of the image respectively, and the value of the array element represents the brightness or color information of the pixel.

[0039] Next, normalization is performed. A normalization algorithm is written in a programming language (such as Python) to divide the pixel value of each frame of the image by 255 so that the pixel value range is between 0 and 1. During the processing, for each pixel in the image, its RGB value is read in turn and normalization calculations are performed separately to ensure the stability of the image features and facilitate subsequent feature extraction operations.

[0040] Preferably, in the sensor data preprocessing, after the sensor data is transmitted to the data preprocessing module, the Kalman filter algorithm is first used for denoising. According to the data type (such as acceleration, velocity, etc.) and acquisition frequency collected by the sensor, the parameters of the Kalman filter, such as the state transfer matrix, observation matrix, process noise covariance matrix and measurement noise covariance matrix, are set. Through iterative calculation, the sensor data is filtered to remove the noise signal caused by environmental interference or sensor errors, and relatively smooth and accurate data is obtained.

[0041] Then the sensor is calibrated according to its initial calibration parameters. When the system is initialized, the known accurate value measured by the sensor under standard environment is compared with the sensor output value to calculate the calibration coefficient. In actual data processing, the collected sensor data is multiplied by the corresponding calibration coefficient to correct the data and ensure data accuracy.

[0042] Finally, normalization is performed to calculate the mean and standard deviation of the sensor data. The mean is subtracted from each data point and divided by the standard deviation to make the data collected by different sensors comparable and mapped to a numerical range between 1 and 1, which is convenient for subsequent multimodal fusion and analysis.

[0043] The feature extraction module is connected to the data preprocessing module, and uses a convolutional neural network to extract the features of the video image preprocessing data to obtain a video image feature vector; wherein the convolutional neural network includes a plurality of convolutional layers that are sequentially connected to learn the posture features of various parts of the athlete's body, and the convolutional layers are sequentially connected to a pooling layer for reducing the data dimension and a fully connected layer for outputting the video image feature vector to the multimodal fusion module. At the same time, the convolutional layers use different convolution kernel sizes, and the pooling layer uses a maximum pooling operation.

[0044] Preferably, in the construction and training of the convolutional neural network (CNN), the construction of the convolutional neural network (CNN) is implemented using a deep learning framework (such as TensorFlow or PyTorch). The first convolution layer uses a 3×3 convolution kernel, 32 in number, a step size of 1, and a padding of 1 to keep the image size unchanged. The convolution kernel slides on the image to perform a convolution operation, and a pixel value in the feature map is obtained by weighted summation. Each convolution kernel generates a feature map, for a total of 32 feature maps. The ReLU activation function is then used to perform a nonlinear transformation on the convolution result to enhance the network's expressive power.

[0045] The pooling layer uses a maximum pooling window of 2×2 with a step size of 2 to downsample the feature map output by the convolutional layer. The maximum value in each pooling window is selected as the corresponding pixel value in the output feature map, reducing the size of the feature map by half, for example, from 224×224 to 112×112, while retaining important features and reducing data dimensions and computation.

[0046] The second convolution layer uses 64 convolution kernels of 5×5 size, with a step size of 1 and a padding of 2, and is also processed by the ReLU activation function. In the future, you can appropriately increase the convolution layer and adjust the convolution kernel parameters according to actual needs and experimental results. For example, add a third convolution layer, use 128 3×3 convolution kernels, etc.

[0047] The fully connected layer integrates the features processed by multiple convolutional layers and pooling layers. First, the feature map output by the pooling layer is flattened into a one-dimensional vector. For example, if the feature map output by the last pooling layer is 7×7×64, the length of the flattened vector is 7×7×64=3136. The fully connected layer contains 128 neurons, which are multiplied by the input vector through the weight matrix and added with the bias term. Then, after the ReLU activation function, a 128-dimensional feature vector is output. The feature vector represents the athlete's body posture features extracted from the video image and will be output to the multimodal fusion module. When training CNN, a large amount of annotated pickleball player action image data is used, and a stochastic gradient descent (SGD) variant optimization algorithm (such as Adagrad) is used. The convolution kernel weights and fully connected layer parameters are adjusted in combination with the back-propagation algorithm so that the model can accurately extract features.

[0048] The multimodal fusion module is connected to the data preprocessing module and the feature extraction module respectively, and is used to fuse the video image feature vector with the sensor preprocessing data to obtain a fused feature vector. The multimodal fusion module fuses the video image feature vector with the sensor preprocessing data by splicing or weighted summation.

[0049] Preferably, after receiving the image feature vector from the feature extraction module and the sensor data from the data preprocessing module, the multimodal fusion module uses a weighted summation method for fusion. After a large number of experiments and data analysis, it is determined that the weight of the image feature vector is 0.6 and the weight of the sensor data is 0.4. In a specific implementation, a fusion algorithm is written in a programming language (such as Python), and each element in the image feature vector is multiplied by 0.6, and each element in the sensor data is multiplied by 0.4, and then the corresponding elements are added to obtain a fused feature vector. For example, if the image feature vector is [0.2, 0.3, 0.4] and the sensor data is [0.5, 0.6, 0.7], the fused feature vector is [(0.2×0.6)+(0.5×0.4), (0.3×0.6)+(0.6×0.4), (0.4×0.6)+(0.7×0.4)]=[0.32, 0.42, 0.52]. The fused feature vector can fully combine the advantages of video image data and sensor data to provide richer and more accurate information for action classification.

[0050] The action classification module, connected to the multimodal fusion module, uses a recurrent neural network to classify the fused feature vector and identify the athlete's action type.

[0051] Preferably, in the construction and training of the recurrent neural network (RNN), the long short-term memory network (LSTM) is preferably used to construct the recurrent neural network (RNN). The LSTM network structure is implemented using a deep learning framework (such as TensorFlow or PyTorch). The input layer receives the fused feature vector output by the multimodal fusion module, and the vector dimension is determined according to the fusion method (such as the sum of the image feature vector dimension and the motion data dimension in the splicing method).

[0052] The forget gate, input gate, cell state update, output gate and other structures inside the LSTM unit are implemented according to standard formulas and algorithms. The forget gate generates a forgetting coefficient through the sigmoid activation function based on the hidden layer state at the previous moment and the current input fusion feature vector, and determines which information in the cell state at the previous moment needs to be forgotten. The input gate also generates an update coefficient through the sigmoid activation function based on the hidden layer state at the previous moment and the current input, and generates new candidate cell state information through the tanh activation function, and the two are combined to update the cell state. The output gate generates an output coefficient through the sigmoid activation function based on the updated cell state and the hidden layer state at the previous moment, and then multiplies the cell state with the output coefficient after being processed by the tanh activation function to obtain the output of the hidden layer at the current moment. When processing a sequence of fused feature vectors, the LSTM unit can retain historical information and model long-term dependencies in action sequences.

[0053] The output layer performs classification prediction through the fully connected layer according to the output of the hidden layer. The number of neurons in the fully connected layer is the same as the number of action categories (for example, forehand shots, backhand shots, moving steps, etc., assuming that there are N categories, then the number of neurons in the fully connected layer is N). The output layer uses the softmax activation function to convert the output into a probability distribution. Each neuron outputs the probability of the corresponding action type, and the category with the highest probability is the predicted action type. During the training process, the Adam optimization algorithm is used, the learning rate is set to 0.001, and the parameters of LSTM are adjusted in combination with the back propagation algorithm. The loss function uses the cross entropy loss function to evaluate and optimize the accuracy of action classification. A large amount of annotated pickleball player action data is used for training, the predicted results are compared with the actual annotations, the error is calculated according to the loss function, and the network parameters are adjusted by back propagation to continuously improve the classification accuracy of the model.

[0054] The motion analysis module is connected to the motion classification module, and analyzes the athlete's motion according to the identified motion type to obtain motion analysis data. The motion analysis module fits the joint motion trajectory by establishing a kinematic model, and calculates the Pearson correlation coefficient between the body posture data and the hitting effect to analyze the correlation; the motion analysis data includes the joint motion trajectory data, the correlation data between the body posture change and the hitting effect, and the division of the motion stages and the extraction of feature data of each stage.

[0055] Preferably, in the joint motion trajectory analysis, according to the recognition result of the action classification module, the corresponding original video image data and motion data are obtained from the data acquisition module. For the identified action type (such as forehand stroke action), the key joints (wrist, elbow, shoulder, etc.) of the athlete's body are selected. The position coordinates of the joints at different time points are calculated using the joint position information in the video image data (through an image processing algorithm, such as obtaining the coordinates of the joints in the image based on feature point detection or deep learning target detection algorithm) and the posture data in the sensor data. For example, for the wrist joint, the pixel coordinates of the wrist joint are located in the video image by the target detection algorithm, and the pixel coordinates are converted into actual space coordinates in combination with the body posture information in the sensor data (such as the rotation angle of the arm, offset, etc.). Then establish a kinematic model, preferably using a combination of trigonometric functions and polynomial functions, such as using a cubic polynomial function to fit the joint motion trajectory (assuming time is t, the joint position coordinates are x(t) = at^3 + bt^2 + ct + d, where a, b, c, d are coefficients to be determined, and the coefficients are fitted according to the joint position coordinate data by the least squares method and other methods), and analyze the laws and characteristics of joint motion, such as the speed of joint motion (via differentiation to obtain v(t) = 3at^2 + 2bt + c), acceleration (derivative again to obtain a(t) = 6at+2b), etc.

[0056] Preferably, in the correlation analysis between body posture and batting effect, the Pearson correlation coefficient between body posture data (such as body inclination angle, degree of twisting, etc.) and batting effect (such as ball flight trajectory, landing point, etc.) is calculated. First, the body posture data is obtained from the sensor data, and the batting effect data is obtained from the video image data (analyzing the ball's movement trajectory and landing point through image processing algorithms) or other auxiliary sensors (such as sensors installed around the venue to monitor the ball's movement trajectory). Then, the Pearson correlation coefficient is calculated using a statistical analysis method. For example, for the body inclination angle and the horizontal displacement of the ball, assuming that the body inclination angle sequence is x=[x1,x2,...,xn], and the ball's horizontal displacement sequence is y=[y1,y2,...,yn], then the Pearson correlation coefficient where x- and are the means of x and y, respectively. By analyzing the correlation coefficient, the influence of body posture on the hitting effect can be evaluated. For example, if the correlation coefficient is high (such as r>0.7), it means that body posture has a greater influence on the hitting effect, and athletes can adjust their posture accordingly.

[0057] Preferably, in the action stage division and feature extraction, the action stages are accurately divided, such as dividing the forehand stroke action into the preparation stage (arm swing back, body center of gravity transfer, etc.), the swing stage (arm accelerated swing, body twisting force, etc.), the moment of hitting the ball (the moment when the ball contacts the racket, analyzing the parameters such as the hitting force and angle), and the follow-up swing stage (the arm continues to swing, the body restores balance, etc.). In the division process, the action posture changes in the video image data (detecting the start, end and key change points of the action through the image processing algorithm) and the speed and acceleration changes in the sensor data (setting a threshold, when the speed or acceleration exceeds the threshold, it is judged as a stage transition point). For each stage, extract feature parameters, such as speed (including joint movement speed, body movement speed, etc., obtained by calculating the position change of adjacent time points divided by the time interval), acceleration (joint acceleration, body acceleration, etc., obtained by calculating the rate of change of speed), joint angle change range (obtained by subtracting the minimum value from the maximum value of the joint angle in this stage), etc., to comprehensively evaluate the action quality.

[0058] The training module is connected to the feature extraction module and the action classification module respectively, and trains the convolutional neural network and the recurrent neural network respectively through the athlete's motion data. Among them, the training module uses the stochastic gradient descent and its variant optimization algorithms, such as the Adagrad algorithm, the Adadelta algorithm or the Adam algorithm, combined with the back propagation algorithm to train the model, and the loss function uses the mean square error or cross entropy loss function.

[0059] Preferably, in data collection and annotation, a large amount of pickleball motion data including different athletes, different hitting methods (such as forehand hitting, backhand hitting, high-flying ball, flat hitting, etc.), and different tactical scenarios (such as offensive and defensive tactics in singles, cooperation tactics in doubles, etc.) are collected. The data source can include game videos, training videos of professional athletes, sports records of amateurs, etc. In the collection process, ensure the diversity and representativeness of the data. Then, manual annotation is performed, and professional pickleball coaches and athletes are organized to annotate the motion data in detail according to their experience and professional knowledge. The annotation content includes information such as action type (such as clearly marked as specific actions such as forehand hitting, backhand hitting, etc.), action stage (accurately dividing the different stages of each action), key technical parameters (such as the racket angle at the moment of hitting, the estimated value of the hitting force, etc.), etc., to construct a rich and comprehensive training data set and test data set, and the ratio of the training data set and the test data set can be set according to the actual situation, such as 8:2 or 7:3.

[0060] Preferably, in model training and evaluation, the training data set is used to train the CNN and RNN models in the system. For the CNN model, the stochastic gradient descent (SGD) variant optimization algorithm Adagrad is used, and the initial learning rate is set to 0.1. According to the change of the loss value during the training process, the learning rate decays to 0.9 times the original after every 10 training rounds. During the training process, the training data is input into the model in batches (such as 64 samples per batch), and the error between the prediction result of the model and the true annotation is calculated (the loss function uses the mean square error (MSE) or the cross entropy loss function, which is selected according to the specific task requirements, such as for multi-classification action type recognition tasks, the cross entropy loss function is more suitable), and the gradient of the model parameters is calculated in combination with the back propagation algorithm, and the model parameters are updated using the optimization algorithm, so that the loss function value of the model on the training data set gradually decreases, thereby improving the accuracy of the model.

[0061] For the RNN model, the Adam optimization algorithm is also used, the learning rate is set to 0.001, the model parameters are adjusted in combination with the back-propagation algorithm, and the cross-entropy loss function is used as the loss function. During the training process, the test data set is used regularly to evaluate the model, and the accuracy (the proportion of correctly predicted samples to the total number of samples), recall (the proportion of correctly predicted positive samples), F1 value (an indicator of comprehensive accuracy and recall, F1 = 2*(accuracy*recall) / (accuracy+recall)) and other evaluation indicators are calculated. When the performance of the model on the test data set does not improve for 5 consecutive rounds (such as the accuracy no longer increases or the F1 value no longer improves) or reaches the preset training rounds (such as 100 rounds), stop training, save the trained model parameters, and use them for actual action recognition and analysis tasks.

[0062] The output module is connected to the motion analysis module and outputs athlete motion recognition results, motion analysis reports and training suggestions based on the motion analysis data.

[0063] Preferably, in the visual display, the system displays the action recognition results in a visual way on the user interface. Using a graphics processing library (such as Python's Matplotlib or OpenCV's visualization function), the athlete's action type and action stage are marked with frame lines of different colors and thicknesses in the video screen. For example, a forehand stroke action is marked with a green frame line, and a backhand stroke action is marked with a red frame line. The thickness of the frame line is adjusted according to the importance or attention of the action. For the action stage, different stages are represented by dotted frame lines or shaded areas of different colors, such as the preparation stage is represented by light blue shadows, and the swing stage is represented by yellow shadows, etc., which is convenient for athletes and coaches to observe and understand intuitively.

[0064] Preferably, in the analysis report generation, a detailed motion analysis report is generated, including contents such as joint motion trajectory diagram, body posture and batting effect correlation analysis chart, action stage characteristic parameter table, etc. For the joint motion trajectory diagram, with time as the horizontal axis and joint position coordinates as the vertical axis, a drawing software (such as Matplotlib) is used to draw a clear curve to show the position change of the joint during the movement. The body posture and batting effect correlation analysis chart uses intuitive forms such as scatter plots, line graphs or bar graphs to display the correlation coefficient, for example, a scatter plot is used to display the correlation between the body tilt angle and the ball landing position, and each point represents the data in a batting action. The action stage characteristic parameter table lists the parameters such as speed, acceleration, angle change, etc. of each stage in detail, and the data in the table retains appropriate decimal places (such as two decimal places), and adds units (such as speed units are m / s, acceleration units are m / s 2 , angle unit is °), and is presented to users in the form of a combination of text and charts, providing a scientific basis for athletes' technical improvement and training plan formulation.

[0065] Preferably, in the generation of training suggestions, the system automatically generates personalized training suggestions based on the results of the action analysis. For example, in response to problems that athletes have in a certain action link, such as incorrect racket angle at the moment of hitting the ball in a forehand shot, the system provides corresponding training methods, such as recommending specific forehand shot angle correction exercises, including setting a fixed shot target, requiring athletes to keep the racket angle within a certain range when hitting the ball. At the same time, according to the overall technical level and training goals of the athletes, reasonable training intensity suggestions are given, such as practicing the action 30-50 times a day, each exercise lasting 10-15 minutes, divided into 3-5 groups, and resting for 1-2 minutes between each group, to help athletes train in a targeted manner and improve their competitive level.

[0066] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A pickleball player motion recognition and analysis system based on multimodal data fusion, characterized by: Included a data acquisition module for collecting video image data of the pickleball player during the exercise and sensor data of the player's movement and the pickleball's movement during the exercise; The data preprocessing module is connected to the data acquisition module, and preprocesses the video image data and the sensor data to obtain the video image preprocessing data and the sensor preprocessing data; The feature extraction module is connected to the data preprocessing module and uses a convolutional neural network to extract the features of the video image preprocessing data to obtain a video image feature vector; The multimodal fusion module is connected to the data preprocessing module and the feature extraction module respectively, and is used to fuse the video image feature vector with the sensor preprocessing data to obtain a fused feature vector; The action classification module is connected to the multimodal fusion module and uses a recurrent neural network to classify the fused feature vectors and identify the athlete's action type; The motion analysis module is connected to the motion classification module, and analyzes the athlete's motion according to the identified motion type to obtain motion analysis data; The training module is connected to the feature extraction module and the action classification module respectively, and trains the convolutional neural network and the recurrent neural network respectively through the athlete's motion data; The output module is connected to the motion analysis module and outputs athlete motion recognition results, motion analysis reports and training suggestions based on the motion analysis data.

2. The pickleball player motion recognition and analysis system based on multimodal data fusion according to claim 1, characterized in that: The data acquisition module includes a camera, a pickle ball motion sensor installed on the pickle ball to collect pickle ball motion data, and an athlete motion sensor worn on the athlete to collect athlete motion data. The camera collects video image data, and the pickle ball motion sensor and the athlete motion sensor collect sensor data.

3. The pickleball player motion recognition and analysis system based on multimodal data fusion according to claim 1, characterized in that: The data preprocessing module performs normalization preprocessing on the video image data to obtain video image preprocessing data; The data preprocessing module performs denoising, filtering and normalization on the sensor data to obtain sensor preprocessing data.

4. The pickleball player motion recognition and analysis system based on multimodal data fusion according to claim 1, characterized in that: The convolutional neural network in the feature extraction module includes multiple convolutional layers that are connected in sequence for learning the posture features of various parts of the athlete's body. The convolutional layers are connected in sequence to pooling layers for reducing data dimensions and fully connected layers for outputting video image feature vectors to the multimodal fusion module.

5. The pickleball player motion recognition and analysis system based on multimodal data fusion according to claim 4, characterized in that: The convolutional layers use convolution kernels of different sizes, and the pooling layer uses the maximum pooling operation.

6. The pickleball player motion recognition and analysis system based on multimodal data fusion according to claim 1, characterized in that: The multimodal fusion module fuses the video image feature vector with the sensor preprocessing data using splicing or weighted summation.

7. The pickleball player motion recognition and analysis system based on multimodal data fusion according to claim 1, characterized in that: The motion analysis module fits the joint motion trajectory by establishing a kinematic model, and calculates the Pearson correlation coefficient between body posture data and batting effect to analyze the correlation; the motion analysis data includes joint motion trajectory data, correlation data between body posture changes and batting effect, as well as dividing the action stages and extracting feature data for each stage.

8. The pickleball player motion recognition and analysis system based on multimodal data fusion according to claim 1, characterized in that: The training module uses stochastic gradient descent and its variant optimization algorithms combined with the back propagation algorithm to train the model.

9. The pickleball player motion recognition and analysis system based on multimodal data fusion according to claim 8, characterized in that: Stochastic gradient descent and its variant optimization algorithms include Adagrad algorithm, Adadelta algorithm or Adam algorithm.

10. The pickleball player motion recognition and analysis system based on multimodal data fusion according to claim 8, characterized in that: The loss function uses mean square error or cross entropy loss function.

Images