Wearable equipment sports action recognition method based on knowledge distillation
Through knowledge distillation technology, complex deep learning models are simplified and deployed on wearable devices, solving the problem of insufficient sports action recognition performance on wearable devices, and achieving efficient action recognition and real-time recommendations, which are suitable for offline environments.
Patent Information
- Application Number
- CN202411285475.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2025-01-07
AI Technical Summary
The prior art is difficult to achieve efficient sports action recognition on wearable devices, due to computing power, battery capacity and storage space, and poor performance in offline environments and strongly rely on image or video data.
Using a knowledge distillation-based method, by training the TabNet teacher model and transferring it to the TabNet student model, complex deep learning models are simplified and deployed on wearable devices, using motion type rule models and biomechanical models for data analysis and action recognition.
It realizes efficient identification of sports movements on wearable devices with limited resources, provides real-time action suggestions and damage warnings, and does not rely on image or video data. It is suitable for offline environments, improving the efficiency and safety of sports training.
Smart Images
Figure CN119272108A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of lightweight deep learning models and sensor data processing, and in particular to a wearable device sports action recognition method based on knowledge distillation. Background Art
[0002] With the rapid development of artificial intelligence technology, its application in various fields is becoming more and more extensive, especially in sports action recognition. At present, AI models deployed on large computing machines can already recognize and analyze human movements with high accuracy, providing strong technical support for athlete training, health monitoring and other scenarios. However, in the field of wearable devices, especially those that need to work in offline environments, existing technologies often face the dual challenges of performance and real-time performance.
[0003] In existing technologies, sports action recognition usually relies on complex machine learning or deep learning algorithms, which require a large amount of data for training and run on high-performance computing platforms to achieve fast responses. However, when these algorithms are applied to wearable devices, they are often difficult to achieve ideal results due to the limitations of the computing power, battery capacity and storage space of the devices.
[0004] Existing technical solutions, such as classifiers or neural network models based on traditional machine learning, can achieve basic action recognition functions in some cases, but their performance in offline environments is often limited by the complexity of the model and the hardware limitations of the device, and they are highly dependent on image or video data.
[0005] Therefore, the prior art has defects and needs to be improved. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a wearable device sports action recognition method based on knowledge distillation.
[0007] The technical solution of the present invention is as follows: a wearable device sports action recognition method based on knowledge distillation is provided, including: a training phase and an inference phase, wherein:
[0008] The specific steps of the training phase are:
[0009] Step a.1: Process all the data stored in the database;
[0010] Step a.2: Use the sports type rule model to determine the sports type to which the sports key data belongs, and send it as a column of data to the TabNet teacher model for training;
[0011] Step a.3: The training process is transferred to the synchronously trained TabNet student model through knowledge distillation;
[0012] Step a.4: Deploy the trained TabNet student model on the wearable devices of several users;
[0013] The specific steps of the reasoning phase are:
[0014] Step b.1: Collecting sports data generated by the user during sports activities through wearable devices;
[0015] Step b.2: Process the data collected by the wearable device in full;
[0016] Step b.3: Use the motion type rule model to determine the motion type of the collected key motion data, input it into the TabNet student model as a column of data, and combine it with the TabNet student model trained in the training phase to perform technical action recognition output;
[0017] Step b.4: The TabNet student model is combined with the biomechanical model to perform biomechanical analysis, action suggestions and planning based on the temporal action trajectory.
[0018] Furthermore, the TabNet teacher model and the TabNet student model are both composed of an encoder and a decoder, and the encoder implements feature selection and processing through a feature transformer and an attention transformer.
[0019] Furthermore, the TabNet teacher model and the TabNet student model use a sparsemax activation function to achieve sparse selection of features.
[0020] Furthermore, in step a.4, the trained TabNet student model is deployed in the wearable devices of different users, and analysis is performed based on the data collected synchronously by multiple wearable devices to provide users with teamwork action analysis suggestions.
[0021] Furthermore, the specific steps of step b.4 are: combining the TabNet student model deployed in the wearable device with the biomechanical model, systematically analyzing the user's motion posture and motion trajectory collected by the wearable device, so as to provide corresponding exercise suggestions for the user's movements or warn of sports injuries.
[0022] Furthermore, the analysis steps of the TabNet student model are as follows: the TabNet student model passes the initial input features through the feature transformer to obtain the initial feature representation, uses the initial feature representation as the input of the attention transformer, selects a feature subset through the attention transformer and passes it to the next step; the TabNet student model generates the final prediction result by using the feature transformer output of each decision step.
[0023] Using the above scheme, the present invention first processes the collected training data in full, and uses the motion type rule model to judge the type of motion of the current data based on key data such as motion frequency, as a column of data, and as a large wide table to train the TabNet teacher model. The TabNet teacher model has the characteristics of a complex multi-layer network and multiple parameters, which can better identify technical movements and effectively train the model. The trained TabNet student model is deployed on the wearable devices of several users. After the deployment of the TabNet student model is completed, in the reasoning stage, only the TabNet student model needs to be used to identify the motion information collected by the wearable device, so as to analyze the motion type and motion data through the TabNet student model, so as to provide corresponding exercise suggestions. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a schematic diagram of the process of the present invention.
[0025] Figure 2 This is the model framework diagram of the TabNet teacher model.
[0026] Figure 3 This is the model framework diagram of the TabNet student model. DETAILED DESCRIPTION
[0027] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0028] See also Figures 1 to 3 The present invention provides a wearable device sports action recognition method based on knowledge distillation, including: a training phase and an inference phase, wherein:
[0029] The specific steps of the training phase are:
[0030] Step a.1: Store the data collected in the past in the database, and perform full data processing on the data in the database, such as splicing wide tables, mean standard deviation and other simple processing.
[0031] Step a.2: Use the exercise type rule model to judge key data such as exercise frequency, determine the type of exercise it belongs to, and send it as a column of data to the TabNet teacher model for training.
[0032] Step a.3: The training process is transferred to the synchronously trained TabNet student model through knowledge distillation. The student model has two simple layers and few parameters. Through the transfer method, the simple grid TabNet student model can better recognize the user's technical actions.
[0033] Step a.4: Deploy the trained TabNet student model on the wearable devices of several users. After the TabNet student model is deployed, in the inference phase, you only need to use the TabNet student model to identify the motion information collected by the wearable device, so as to analyze the motion type and motion data through the TabNet student model to provide corresponding motion suggestions.
[0034] The trained TabNet student model is deployed in the wearable devices of different users, and the data collected by multiple wearable devices are analyzed to provide users with teamwork action analysis suggestions. When performing team sports, multiple wearable devices are used to collect sports data of different users at the same time, so as to analyze the cooperation between team members in a data-based way, and provide corresponding cooperation action planning and training plans for the team's cooperation.
[0035] The specific steps of the reasoning phase are:
[0036] Step b.1: Use wearable devices to collect motion data generated by users during sports activities, including running, jumping and other movements of users during sports activities. The accelerometer and gyroscope in the wearable device are used to capture and collect the user's motion parameters.
[0037] Step b.2: Process the data collected by the wearable device in full. Process the motion parameter data collected by the wearable device to analyze and identify the motion state.
[0038] The logic of some structured data sources is: Take the accelerometer and gyroscope 3-axis raw signals tAcc-XYZ and tGyro-XYZ. These time domain signals (the prefix "t" represents time) are captured at a constant rate of 50Hz. They are then filtered using a median filter and a 3rd-order low-pass Butterworth filter (with a corner frequency of 20Hz) to remove noise. Similarly, another low-pass Butterworth filter with a corner frequency of 0.3Hz is then used to separate the acceleration signal into body and gravity acceleration signals (tBodyAcc-XYZ and tGravityAcc-XYZ). The linear acceleration and angular velocity of the body are time-derived to obtain Jerk signals (tBodyAccJerk-XYZ and tBodyGyroJerk-XYZ), and the amplitudes of these three-dimensional signals (tBodyAccMag, tGravityAccMag, tBodyAccJerkMag, tBodyGyroMag, tBodyGyroJerkMag) are calculated using the Euclidean norm. Finally, a fast Fourier transform (FFT) is applied to some of these signals to produce fBodyAcc-XYZ, fBodyAccJerk-XYZ, fBodyGyro-XYZ, fBodyAccJerkMag, fBodyGyroMag, fBodyGyroJerkMag. (Note that "f" indicates frequency domain signal).
[0039] Step b.3: Use the motion type rule model to determine the motion type of the collected key motion data, input it into the TabNet student model as a column of data, and combine it with the TabNet student model trained in the training phase to perform technical action recognition output. The collected motion parameter data is analyzed and identified through the motion type rule model to obtain the user's motion type (such as running, jumping, etc.), real-time running speed, moving distance and other data during the current motion process.
[0040] Step b.4: The TabNet student model is combined with the biomechanical model to perform biomechanical analysis, action suggestions and planning for the time-series motion trajectory. The specific steps are: the TabNet student model deployed in the wearable device is combined with the biomechanical model to systematically analyze the user's motion posture and motion trajectory collected by the wearable device, so as to provide corresponding motion suggestions for the user's motion or warn of sports injuries. According to the analysis results of the motion type rule model and the motion parameter data collected by the wearable device, detailed motion data is obtained to identify and analyze the user's motion posture and motion trajectory during the exercise, infer the joint force and motion trajectory, and provide scientific action suggestions for users to help users prevent sports injuries and effectively optimize training results.
[0041] In the present invention:
[0042] Training phase: refers to the process of the model learning to identify different types of movements. During this process, the TabNet teacher model will be provided with a large amount of labeled training data and learn how to identify the correct type of movement from it.
[0043] Inference phase: Once the model is trained, it can enter the inference phase. In this phase, the model uses what it has learned to predict the motion type of new, unseen data. This is the phase where the model is deployed to make predictions in real-world applications.
[0044] TabNet Teacher Model / TabNet Student Model: The model consists of an encoder and a decoder. The encoder implements feature selection and processing through a feature transformer and an attention transformer. The model uses the sparsemax activation function to achieve sparse feature selection, which helps the model focus its learning capacity on the most significant features.
[0045] Motion type rule model: A simple model based on predefined rules, used to identify patterns or features in data. The detection results of the accelerometer and gyroscope are analyzed by the motion type rule model to identify and determine the corresponding motion type.
[0046] Knowledge distillation: A compression technique whereby the knowledge from the TabNet teacher model is transferred to a simpler TabNet student model with fewer parameters. This usually involves adjusting the TabNet student model so that its output is as close as possible to the output of the TabNet teacher model.
[0047] Technical action recognition output: refers to the final prediction result of the model, that is, the type of movement or technical action identified after analyzing the input data (such as data collected by wearable devices). For the time-series action trajectory, biomechanical analysis, action suggestions and planning are carried out.
[0048] Through the above process, the knowledge of a complex deep learning model (TabNet teacher model) can be effectively transferred to one or more simpler models (TabNet student models), so that it can run efficiently on resource-constrained devices (such as wearable devices). In this way, the user's movements can be analyzed in real time during the user's exercise, and more reasonable and effective training or exercise method recommendations can be made.
[0049] The analysis steps of the TabNet student model are as follows: The TabNet student model passes the initial input features through the feature transformer to obtain the initial feature representation. The initial feature representation is used as the input of the attention transformer, and a feature subset is selected by the attention transformer to pass to the next step. There will be a hyperparameter to set the number of times this step is repeated. The TabNet student model generates the final prediction result by using the feature transformer output of each decision step. In addition, the mask of attention is added at each step to understand which features are used for prediction. These masks can be used to obtain local feature importance as well as global importance. Among them:
[0050] Feature Transformer (FT): This is a key component in the TabNet model, responsible for processing input features and generating useful feature representations. This is achieved through a series of fully connected layers (also called dense layers) and batch normalization. The output of the feature transformer will be used in the subsequent attention transformer.
[0051] Attention Transformer (AT): This component is responsible for selecting the most important subset of features at each decision step. Feature selection is achieved by applying a sparsemax activation function so that the model only focuses on the most relevant features at each step. This selective attention helps improve the efficiency and interpretability of the model.
[0052] Feature subset: A set of features selected by the attention transformer at each decision step. These features are considered to be the most useful for the current task, and the model will focus on these features for the next step of learning and prediction.
[0053] Hyperparameters: These are parameters that need to be set before the model is trained, controlling the behavior and complexity of the model. In the TabNet model, an important hyperparameter is the number of steps, which determines feature selection and feature processing. It determines the number of times the feature transformer and attention transformer are repeatedly applied.
[0054] Decision step: Each step in the TabNet model, including feature selection and feature processing. Each decision step generates a new feature representation, which will eventually be combined for prediction.
[0055] Attention Mask: At each decision step, the attention transformer generates a mask indicating which features are selected for prediction at that step. These masks can be used to analyze the local and global feature importance of the model.
[0056] Local feature importance: refers to the features that the model considers important at a specific decision step. By analyzing the attention mask, we can understand which features have a greater impact on the model's prediction at each step.
[0057] Global feature importance: refers to the features that the model considers important in all decision steps of the entire model. By aggregating the attention masks of all steps, the overall importance of each feature in the entire model can be evaluated.
[0058] In the embodiment provided by the present invention, the collected training data is firstly processed in full, and the key data such as the movement frequency are judged as the type of movement of the current data by using the movement type rule model, as a column of data, and as a large wide table, the TabNet teacher model is trained together. The TabNet teacher model has the characteristics of a complex multi-layer network and multiple parameters, which can better identify technical actions and effectively train the model. For example, the sports data related to football are input into the training data, the movement type is identified by the movement type rule model, and the movement information in the training data is identified and analyzed by the TabNet teacher model, so as to train the model by the training data. Then the trained model is transferred to the wearable devices of each team member participating in the training through the knowledge distillation method for deployment. When the team members wear the wearable device for training, the wearable device receives the corresponding motion signal through the built-in accelerometer, gyroscope and other sensor devices, and sends it to the control system of the wearable device, and the motion signal is identified by the movement type rule model, and then the detected data signal is analyzed by the TabNet student model, so as to obtain the motion trajectory of the current team member and the motion specification in the comprehensive database. The model can be used to judge the actions of the players, predict their action tendencies and movement postures, infer the force on the players' joints during the exercise, and issue warnings to the players when possible sports injuries are detected. At the same time, suggestions can be made for the players' actions based on their movement postures and movement trajectories to improve the training effect.
[0059] In addition, by configuring health management modules such as heart rate monitoring in wearable devices, we can propose corresponding breathing and exercise adjustment strategies to the players based on the physical condition of the players during exercise, thereby providing effective action suggestions to the players who are training with wearable devices, improving training efficiency and ensuring the health of the players during exercise.
[0060] Among team members, effective suggestions can be made on team cooperation through the movement trajectories and postures of different team members, and corresponding team tactics can be quickly given according to the postures, relative positions, physical conditions, etc. of different team members, so as to improve the efficiency of team collaboration.
[0061] The present invention does not rely on images or videos, and can perform sports technical action recognition only by analyzing structured data. In addition, due to the use of rule models combined with lightweight deep learning models, it is ensured that the specific technical actions of the user in the sports process can still be recognized in the wearable device in an offline state. By combining the biomechanical model, the system analyzes the user's movement posture and movement trajectory, infers the joint force conditions and movement trajectory, and thus provides scientific action suggestions for the user, helping the user to prevent sports injuries and optimize training results. In addition, through the synchronous data collection and analysis of multiple wearable devices, complex action recognition in team sports is realized, so that the system can better recognize team cooperation actions, which is particularly suitable for action recognition and analysis in team sports such as basketball and football.
[0062] In summary, the present invention first processes the collected training data in full, and uses the motion type rule model to judge what type of motion the current data is based on key data such as motion frequency, as a column of data, and as a large wide table to train the TabNet teacher model. The TabNet teacher model has the characteristics of a complex multi-layer network and multiple parameters, which can better identify technical movements and effectively train the model. The trained TabNet student model is deployed on the wearable devices of several users. After the deployment of the TabNet student model is completed, in the reasoning stage, only the TabNet student model needs to be used to identify the motion information collected by the wearable device, so that the motion type and motion data are analyzed through the TabNet student model to provide corresponding exercise suggestions.
[0063] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A wearable device sports action recognition method based on knowledge distillation, characterized in that: include: Training phase and inference phase, where: The specific steps of the training phase are: Step a.1: Process all the data stored in the database; Step a.2: Use the sports type rule model to determine the sports type to which the sports key data belongs, and send it as a column of data to the TabNet teacher model for training; Step a.3: The training process is transferred to the synchronously trained TabNet student model through knowledge distillation; Step a.4: Deploy the trained TabNet student model on the wearable devices of several users; The specific steps of the reasoning phase are: Step b.1: Collecting sports data generated by the user during sports activities through wearable devices; Step b.2: Process the data collected by the wearable device in full; Step b.3: Use the motion type rule model to determine the motion type of the collected key motion data, input it into the TabNet student model as a column of data, and combine it with the TabNet student model trained in the training phase to perform technical action recognition output; Step b.4: The TabNet student model is combined with the biomechanical model to perform biomechanical analysis, action suggestions and planning based on the temporal action trajectory.
2. The wearable device sports action recognition method based on knowledge distillation according to claim 1 is characterized in that: The TabNet teacher model and the TabNet student model are both composed of an encoder and a decoder. The encoder implements feature selection and processing through a feature transformer and an attention transformer.
3. The wearable device sports action recognition method based on knowledge distillation according to claim 1 is characterized in that: The TabNet teacher model and the TabNet student model use the sparsemax activation function to achieve sparse selection of features.
4. The wearable device sports action based on knowledge distillation method according to claim 1, characterized in that: In step a.4, the trained TabNet student model is deployed in the wearable devices of different users, and the data collected synchronously by multiple wearable devices is analyzed to provide users with teamwork action analysis suggestions.
5. The wearable device sports action recognition method based on knowledge distillation according to claim 1 is characterized in that: The specific steps of step b.4 are: combining the TabNet student model deployed in the wearable device with the biomechanical model, systematically analyzing the user's motion posture and motion trajectory collected by the wearable device, so as to provide corresponding motion suggestions for the user's motion or warn of sports injuries.
6. The wearable device sports action recognition method based on knowledge distillation according to claim 1 or 5, characterized in that: The analysis steps of the TabNet student model are as follows: the TabNet student model passes the initial input features through the feature transformer to obtain the initial feature representation, uses the initial feature representation as the input of the attention transformer, selects a feature subset through the attention transformer and passes it to the next step; the TabNet student model generates the final prediction result by using the feature transformer output of each decision step.