Wearable fitness action recognition and index calculation method and device

Abstract

The invention discloses a wearable fitness action recognition and index calculation method and device. The method comprises the following steps: acquiring motion data of a target user by using wearable equipment; inputting the data into a motion recognition module to judge that the user is in a motion state at present; under the condition that the target user is judged to be in the motion state, inputting corresponding data into a trained motion recognition model so as to judge the type of the motion which is being performed by the target user; and under the condition that the action category is judged to be correct, inputting corresponding data into a calculation module to calculate to obtain a corresponding index, and outputting a calculation result. According to the method, the motion state and the motion type are recognized by using the motion data in sequence, and the result is calculated and output on the basis, so that the user can know own motion in detail.

Description

Technical Field

[0001] The present invention belongs to the technical field of wearable devices, and in particular relates to a wearable fitness action recognition and index calculation method and device. Background Art

[0002] With the rapid development of my country's economy and the increasing needs of the people in all aspects, some elite fitness enthusiasts and professional athletes have a growing demand for scientific and effective fitness. Although the ways of choosing fitness and exercise are different at this stage, in order to pursue a good figure and body shaping, more people will choose strength training.

[0003] In today's fitness industry, strength training is mainly achieved through linear sensors. They can capture subtle changes in movements, but their use is restricted by reasons such as inconvenience in carrying and inability to monitor the way force is exerted. When obtaining indicators of force exertion, force platforms and pressure gauges are currently used. Most of these instruments are laboratory equipment and can obtain precise values, but they also have problems such as inconvenience in carrying and inability to calculate results in real time.

[0004] Wearable devices provide a lower cost and more portable wearing method. At the same time, through the internal integration of IMU, barometer and other sensors, they can collect the user's training data in real time, and provide real-time data feedback through indicator calculation methods, thereby realizing data calculation and display regardless of location and time, making it easier for users to wear and use. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a wearable fitness action recognition and index calculation method and device.

[0006] According to a first aspect of the present invention, a wearable fitness action recognition and index calculation method is provided. The method comprises the following steps:

[0007] Use wearable devices to collect target users’ sports data;

[0008] Inputting the motion data into a motion recognition module to determine whether the target user is in a motion state;

[0009] In the case of being determined as a motion state, the corresponding motion data is input into the trained motion recognition model to determine the type of motion the target user is performing:

[0010] When the action category is determined to be correct, the corresponding motion data is input into the calculation module to calculate the corresponding index and output the calculation result;

[0011] According to a second aspect of the present invention, a wearable fitness action recognition and index calculation device is provided; the device comprises:

[0012] Wearable devices: used to obtain the target user’s sports data;

[0013] Motion recognition module: used to input the motion data into the motion recognition module to determine whether the target user is in a motion state;

[0014] Action recognition module: when the user is in motion, the corresponding motion data is input into the trained action recognition model to determine the type of action the target user is performing:

[0015] Calculation module: when the action category is determined to be correct, the corresponding data is input into the calculation module to calculate the corresponding index and output the calculation result;

[0016] Compared with the prior art, the advantage of the present invention is that the provided motion recognition and index calculation method is implemented on a wearable device, and the wearable device is used to automatically collect motion data, and the motion data is passed to the motion recognition model to obtain the recognition result, and then the motion data identified as the correct action is passed to the calculation module to calculate the corresponding index for the completion of the action, so as to understand the user's action completion status in real time, make exercise suggestions in time, and effectively improve exercise performance.

[0017] The invention has reasonable design, is simple and convenient to wear, can detect the user's exercise behavior without affecting the exercise state, and is easy to promote and implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0019] Figure 1 is a flow chart of a wearable fitness action recognition and index calculation method according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of the overall process of a wearable fitness action recognition and index calculation method according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of a framework of a wearable device according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of the appearance of a wearable device according to an embodiment of the present invention;

[0023] Figure 5is a schematic diagram of the internal structure of a wearable device according to an embodiment of the present invention;

[0024] Figure 6 is a schematic diagram of an action recognition network structure according to an embodiment of the present invention;

[0025] Figure 7 is a schematic diagram of a process of training an action recognition model according to an embodiment of the present invention;

[0026] Figure 8 is a schematic diagram of a process of performing motion recognition and index calculation using a wearable device according to an embodiment of the present invention;

[0026] Fig. 9 is a schematic diagram of recognition accuracy of different target user actions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the components and steps described in these embodiments do not limit the scope of the present invention.

[0028] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0029] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0030] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0031] In general, the present invention uses a portable wearable device to collect the user's motion data and determine their motion state, thereby realizing motion recognition and index result calculation. Due to its storage feature, the present invention can realize data collection regardless of scene or time, reduce the cost of data collection, better obtain data of users in different motion states, and thus improve the robustness of the motion recognition model.

[0032] See also Figure 1As shown, the provided wearable fitness action recognition and index calculation method includes: step S01, using a wearable device to collect the motion data of the target user; step S02, inputting the motion data into a motion recognition module to determine whether the target user is in a motion state; step S03, when it is determined to be in a motion state, inputting the corresponding data into a trained action recognition model to determine the action category of the target user; step S04, when it is determined that the action category is correct, inputting the corresponding data into a calculation module to calculate the corresponding index and output the calculation result.

[0033] Correspondingly, the present invention also provides a wearable fitness action recognition and index calculation device. The device includes: a wearable device, which is used to obtain the motion data of the target user; a motion recognition module, which is used to input the motion data into the motion recognition model to determine whether the target user is in a motion state; the action recognition module, when it is determined to be in a motion state, inputs the corresponding motion data into the trained action recognition model to determine the action category of the target user; and a calculation module: when it is determined that the action category is correct, inputs the corresponding data into the calculation module to calculate the corresponding index and output the calculation result.

[0034] The motion recognition module and action recognition module described therein can be embedded in a wearable device or be independent of the wearable device. For example, the functions of the motion recognition module and the action recognition module are integrated into the wearable device, or integrated into a mobile terminal, a server or a cloud. In the following, the example of embedding the motion recognition module and the action recognition module in a wearable device is mainly used to illustrate that the portability of the wearable device can be fully utilized to expand the application of the wearable device.

[0035] Combination Figure 2 As shown, the scheme of the present invention generally includes two parts: a wearable device and a model algorithm. Researchers can train motion recognition models using their own collected data sets and import these models into wearable devices. Then, the wearable device collects the user's air pressure signal and IMU signal data and makes a motion state judgment, and then performs motion recognition and index calculation. Finally, the wearable device sends the data to the terminal device or background through the embedded wireless module (WIFI or Bluetooth), and the terminal or background can display the user's motion index data.

[0036] 1. Examples of wearable devices

[0037] Figure 3 This is a schematic diagram of the wearable device framework. The device includes an STM32 main control board, multiple sensors, LED lights, a power supply, etc.

[0038] In one embodiment, the connection and communication mode between the STM32 main control board and the multiple sensors is set as follows:

[0039] The STM32 main control board and IMU sensor use the SPI bus to communicate bidirectionally with the IMU sensor through the NSS, CLK, MISO and MOSI pins to read the acceleration, gyroscope and magnetometer data;

[0040] The STM32 main control board and the air pressure sensor use the I2C bus to communicate with the air pressure sensor through the SCL and SDA pins to read the pressure and temperature data;

[0041] The STM32 main control board and LED light use the GPIO port to communicate with the LED light and send commands;

[0042] The STM32 main control board and the wireless module use the GPIO port and the UART serial port to communicate asynchronously with the wireless module to send and receive wireless data.

[0043] Figure 4 This is the overall appearance of the wearable device, which is worn on the wrist or ankle with a matching strap.

[0044] Figure 5 This is the internal structure diagram of the wearable device. The device is embedded with IMU sensors and air pressure sensors, which are connected to the main control board to obtain data information. Through the algorithm model embedded in the main control board, the displacement, speed, force, power, concentric time, and eccentric time of completing an action are calculated, and sent to the mobile terminal or background through the wireless module, and then the data is displayed.

[0045] 2. Implementation example of training action recognition model.

[0046] Figure 6 This is a schematic diagram of the action recognition network structure. The model generally includes a feature extractor and a category classifier. The input motion data includes: acceleration data and gyroscope data measured by the IMU. The two-dimensional data is converted into one-dimensional data through the Linear layer, and then the converted data passes through the Dense layer to reduce the number of features and the number of model parameters. Finally, the features extracted by the feature extractor are sent to the category classifier composed of the Dropout layer, the Dense layer and the Softmax output layer to obtain the probability of each category.

[0047] Figure 7 Schematic diagram of the training process of the action recognition model. In one embodiment, supervised learning is used to solve the action recognition task. The overall training process is a two-stage pre-training-fine-tuning process, and then the fine-tuned model can be used for testing or actual scenarios.

[0048] Specifically, in the pre-training stage, the training data is used to train the feature extractor and the category classifier, and the multi-classification cross entropy loss function is used to evaluate the accuracy of the classification. The pre-trained feature extractor has learned the feature information of the test data, and the category classifier also has a good classification ability for the action recognition task. However, for the verification data, the pre-trained feature extractor has not learned the data features of the verification data users. The verification data is labeled and the model is fine-tuned using supervised learning. In the fine-tuning stage, the training data in the training stage is used as a class of labels, and the verification data is used as a class of labels, and the feature extractor and classifier in the pre-training stage are fine-tuned. Finally, the feature extractor and classifier in the fine-tuning stage can be used to form an action recognition model.

[0049] Fig. 9 This is a schematic diagram of the process of motion recognition using wearable devices. First, the motion recognition model is embedded in the wearable device, and the wearable device can normally collect IMU data and air pressure data. Then, the user performs relevant motion training, and the embedded software first determines whether the user is in motion. When it is determined that the user is in motion, the set of data is passed to the embedded motion recognition model and judged. If the action category is judged incorrectly, it returns to continue collecting user motion data; if the judgment is correct, the result is calculated and output, and the next set of data is received for judgment.

[0050] It should be noted that, without violating the spirit and scope of the present invention, those skilled in the art may make appropriate changes to the above embodiments. For example, relevant actions adapted to their own research field may be collected through wearable devices, and model training may be performed, and the trained model may be embedded in the wearable device to perform recognition and detection of more actions.

[0051] To further verify the effect of the present invention, experiments on action recognition of different groups of people were conducted. For the action recognition task, 27 international athletes, 23 athletes, 17 first-level athletes, 20 second-level athletes, 18 third-level athletes, 25 elite enthusiasts, 18 ordinary enthusiasts, and 15 beginners were used to complete four groups of actions, namely squat, bench press, deadlift, and vertical jump, to test the accuracy of the model. The experimental results show that the accuracy of the models for different groups of people is: 96.5% for international athletes, 97.84% for athletes, 93.78% for first-level athletes, 91.08% for second-level athletes, 86.22% for third-level athletes, 92.16% for elite enthusiasts, 84.32% for ordinary enthusiasts, and 74.05% for beginners. Fig. 9 It is a schematic diagram of the accuracy of the action recognition model.

[0052] In summary, compared with the prior art, the present invention has the following advantages:

[0053] 1) The present invention combines wearable devices and model algorithms, uses wearable devices to collect IMU signal data and air pressure signal data of user movements and determines whether the user is in motion, then passes the data of the motion state into the motion recognition model to detect the action category, and finally passes the data to the calculation module, calculates the corresponding indicators and sends the results through the wireless module.

[0054] 2) In the action recognition task, the present invention improves the recognition ability of the model by a supervised transfer learning-based method to realize the function of user action recognition. When a new user uses the wearable device, the new user can be better adapted so that the new user can understand the completion of the action in real time.

[0055] 3) When training the model, the present invention mainly uses athletes and enthusiasts as training data, and the recognition accuracy for this group of people is relatively high. For beginners, it is recommended to conduct action training under the guidance of a coach.

[0056] Some embodiments of the present invention have been described above. The above description is exemplary and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A wearable fitness action recognition and index calculation method, comprising the following steps: Step 1: Use wearable devices to collect target user's sports data; Step 2: Input the motion data in step 1 into the motion recognition module to determine whether the target user is in motion; Step 3: When the target user is determined to be in a motion state, the corresponding data is input into a trained motion recognition model to determine the type of motion being performed by the target user; Step 4: When the action category is determined to be correct, the corresponding data is input into the calculation module to calculate the corresponding indicators and output the calculation results.

2. The method according to claim 1, characterized in that The motion recognition module is a function based on height change judgment, and its implementation method is as follows: Based on the change of the height value in the motion data, when the height continues to increase or decrease within a certain period of time, it is preliminarily considered that the user is currently in motion.

3. The method according to claim 1, characterized in that The action recognition model includes a feature extractor and a category classifier, and is trained according to the following steps: Based on the set multi-classification cross entropy loss function (categorical_crossentropy), the feature extractor and classifier are pre-trained using the training data to obtain the pre-trained feature extractor and classifier; The training data used in the training process is used as one type of label, and the verification data is used as another type of label, and the parameters of the trained feature extractor and classifier are adjusted to obtain a feature extractor after the parameters are adjusted, wherein in the parameter adjustment process, the loss value is obtained by using the loss function, and a gradient update is performed; The model is composed of the feature extractor and the classifier after parameter adjustment as the trained action recognition model.

4. The method according to claim 3, characterized in that The feature extractor includes a flattening layer and a fully connected layer, and the classifier includes a Dropout layer and a Softmax layer. The flattening layer is used to splice the input motion data into one dimension, and then pass the spliced ​​data into the fully connected layer to obtain the fused features of the input data, and then pass the fused features to the classifier.

5. The method according to claim 1, characterized in that The motion data includes inertial measurement unit signal data IMU and air pressure signal data. Since we need to use altitude data in actual use, we also need to convert the air pressure signal data into altitude. The basic formula is: in: h is the altitude in meters, T0 is the absolute temperature at sea level in Kelvin, L is the coefficient by which the temperature varies with altitude in Kelvin / meter, P0 is the atmospheric pressure at sea level in Pascals, P is the atmospheric pressure at the current location in Pascals, and R is the gas constant.

6. The method according to claim 1, characterized in that The calculation module is a calculation function for calculating the corresponding indicators of the motion process, wherein the corresponding indicators include: displacement height, average concentric velocity, maximum concentric velocity, average concentric power, maximum concentric power, concentric time, centrifugal time, centrifugal speed, maximum centrifugal speed, frontal offset, and lateral offset, wherein: When the movement height value changes from low to high, it is considered to be centripetal movement; when the movement height value changes from high to low, it is considered to be eccentric movement; the specific usage scenario needs to be adjusted according to the actual movement, which will not be elaborated here. The method for calculating the displacement height is to obtain the maximum value of the centroid movement process and subtract the minimum value of the process height; The centripetal time is calculated by subtracting the timestamp corresponding to the maximum height of the centripetal motion process from the timestamp corresponding to the minimum height of the centripetal motion process. The centripetal average velocity is calculated by taking the displacement height and dividing it by the centripetal time; The maximum centripetal velocity is calculated by dividing the height change of every ten frames of data by the time corresponding to the ten frames, and retaining the maximum value of the centripetal motion process; The calculation method of the centripetal average power is: Where m is the user's weight, is the average acceleration of the centripetal motion process, obtained through IMU, is the mean centripetal velocity; The maximum centripetal power calculation method is: m·a max ·v max , where m is the user's weight, a max is the maximum acceleration of the centripetal motion process, obtained through IMU, v max is the maximum centripetal velocity; The centrifugal time is calculated by taking the timestamp corresponding to the maximum value of the centrifugal motion process and subtracting the timestamp corresponding to the minimum value of the centrifugal motion process; The centrifugal speed is calculated by taking the displacement height and dividing it by the centrifugal time; The maximum centrifugal speed is calculated by dividing the height change of every ten frames of data by the time corresponding to the ten frames, and retaining the maximum value of the value during the centrifugal motion process; The front offset is calculated as follows: Where gyroX is the data of the X-axis of the gyroscope in the IMU, and the front offset is obtained by the accumulated value of gyroX during the centripetal motion; The side offset is calculated as follows: Where gyroY is the data of the Y-axis of the gyroscope in the IMU, and the lateral offset is obtained by the accumulated value of gyroY during the centripetal motion.

7. A wearable fitness action recognition and index calculation device, comprising: Wearable devices: used to obtain the target user’s sports data; Motion recognition module: used to input the motion data into the motion recognition model to determine whether the target user is in motion; Action recognition module: used to input the corresponding motion data into the trained action recognition model to determine the action category of the target user when the target user is in motion state; Calculation module: when the action category is determined to be correct, the corresponding data is input into the calculation module to calculate the corresponding indicators and output the calculation results.

8. The device according to claim 7, characterized in that The motion recognition module, the action recognition module and the calculation module are embedded in a wearable device, wherein the wearable device includes an STM32 main control board, an IMU sensor, an air pressure sensor, a FLASH flash memory storage, an LED light, and a wireless module, wherein: The STM32 main control board and IMU sensor use the SPI bus to communicate bidirectionally with the IMU sensor through the NSS, CLK, MISO and MOSI pins to read the acceleration, gyroscope and magnetometer data; The STM32 main control board and the air pressure sensor use the I2C bus to communicate with the air pressure sensor through the SCL and SDA pins to read the pressure and temperature data; The STM32 main control board and FLASH flash memory use the SPI bus to communicate with the memory through the IO3, IO2, IO1, IO0, CLK and NSS pins to store the motion data during the motion process; The STM32 main control board and LED light use the GPIO port to communicate with the LED light and send commands; The STM32 main control board and the wireless module use the GPIO port and the UART serial port to communicate asynchronously with the wireless module to send and receive wireless data.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images