A method and system for identifying and counting movements in training without equipment

Abstract

The present invention discloses a method and system for identifying and counting non-equipment training actions, which belongs to the field of sports health information technology, including collecting training video images of trainees; using a BlazePose network model to process the training video images to obtain a number of feature data; using a motion recognition model based on a semantically guided neural network to perform motion recognition on the several feature data to obtain the training action recognition results of the trainees; using a counting algorithm to count the number of corresponding peaks and troughs in the several feature data to obtain the number of training actions of the trainees. The present invention can be directly run on a mobile phone, does not rely on other devices, and can be combined with a non-equipment training method to effectively supervise and self-check the trainees' movements.

Description

Technical Field

[0001] The present invention relates to the field of sports health information technology, and in particular to a method and system for identifying and counting equipment-free training movements. Background Art

[0002] There are currently two common ways to exercise. The first is professional sports venues such as gyms, but modern people lack the time to complete training in these professional venues due to work or study pressure. The second is fitness software such as Keep, which can complete training in fragmented time without the help of any sports equipment through voice broadcasts and video guidance. However, this training method lacks action recognition function during the training process, and it is impossible to know whether the trainee has completed the training action or whether the training action is standard. It lacks supervision and requires the trainee to manually calculate the number of completed actions, which lacks self-checking. Therefore, it is necessary to explore a new training supervision and self-checking method to make up for the shortcomings of traditional training methods.

[0003] Human motion recognition has been one of the hot research directions in recent years. Currently, there are two main methods for human motion recognition. The first method is to obtain human motion information based on wearable devices to complete motion recognition. However, the use of wearable devices will bring many inconveniences to the human body, which is not conducive to promotion and practical use. The second method is to perform motion recognition based on images. The most typical case is to use Microsoft's Kinect device to complete motion recognition. Although the Kinect-based motion recognition method solves the inconvenience caused by wearable devices to the human body, it is not portable because it relies on Kinect devices and computers. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the above-mentioned background technology and realize motion recognition and counting without equipment.

[0005] To achieve the above objectives, on the one hand, a method for identifying and counting motions without equipment training is adopted, comprising:

[0006] Collecting training video images of trainees;

[0007] The BlazePose network model is used to process the training video images to obtain some feature data;

[0008] The action recognition model based on semantic guided neural network is used to perform action recognition on several feature data to obtain the training action recognition results of the trainee;

[0009] A counting algorithm is used to count the number of corresponding peaks and troughs in a number of characteristic data to obtain the number of training movements of the trainee.

[0010] Furthermore, the training video image is processed using the BlazePose network model to obtain several feature data, including:

[0011] Each frame of the training video image is transformed and then input into the BlazePose network model to obtain the skeleton point data of the human body in each frame of the image;

[0012] According to the characteristics of the training task, feature extraction is performed on the human skeleton point data to obtain the plurality of feature data.

[0013] Furthermore, the feature extraction of human skeleton point data according to the characteristics of the training task to obtain the plurality of feature data includes:

[0014] For the rope skipping and squatting movements, the change in the y-axis coordinate of the nose is used as feature f1: f1 = y nose -y start_nose , assuming that the coordinates of the nose at the beginning of the action and the coordinates of the nose at the current moment are (x start_nose ,y start_nose ) and (x nose ,y nose );

[0015] For the squat action, the vertical difference between the knee and the foot is used as feature f2: f2 = y knee -y foot , assuming that the coordinates of the left knee and foot are (x knee ,y knee ) and (x foot ,y foot );

[0016] In the jumping jack movement, the difference between the hand periodically passing above and below the nose is used as feature f3: f3 = y hand -y nose ,y hand Represents the ordinate of the hand, y nose represents the vertical coordinate of the nose;

[0017] During the movement, the angle between the shoulder and the elbow is used as feature f4: Assume that the coordinates of the shoulder and elbow joints are (x shoulder ,y shoulder ) and (x elbow ,y elbow ).

[0018] Furthermore, the action recognition model based on the semantically guided neural network includes a joint-level module, a frame-level module and a fully connected layer. The action recognition model based on the semantically guided neural network is used to perform action recognition on a number of feature data to obtain a training action recognition result of the trainee, including:

[0019] transforming the plurality of feature data into dynamic representations and inputting them into a joint-level module;

[0020] The joint-level module learns the dynamic representation of skeletal joints by fusing the position and velocity information of skeletal joints and models the dependencies of joints.

[0021] The frame-level module aggregates the output of the joint-level module to obtain all joint information of each frame, and models the dependency between frames to obtain joint information of all frames.

[0022] The fully connected layer obtains the trainee's action based on the output recognition of the frame-level module.

[0023] Furthermore, the joint-level module learns the dynamic representation of skeletal joints by fusing the position and velocity information of skeletal joints and models the dependency of joints, including:

[0024] Connecting the dynamic representation of the plurality of feature data with the bone joint point type to obtain joint data;

[0025] A semantically guided neural network is used to process joint data and model the dependencies between joints.

[0026] Furthermore, the frame-level module aggregates the output of the joint-level module to obtain all joint information of each frame, and models the dependency between frames to obtain joint information of all frame images, including:

[0027] The output of the joint level module is added to the frame sequence, and all joint information in each frame is aggregated using a spatial maximum pooling layer to obtain an aggregated result;

[0028] A convolutional neural network is used to model the dependencies between frames.

[0029] Furthermore, the identified and counted training movements include at least arm extension, jumping jacks, squats and rope skipping.

[0030] On the other hand, a non-equipment training motion recognition and counting system is used, including: an image acquisition module, a feature extraction module, a motion recognition module and a counting module, wherein:

[0031] The image acquisition module is used to acquire the training video images of the trainee;

[0032] The feature extraction module is used to process the training video image using the BlazePose network model to obtain a number of feature data;

[0033] The action recognition module is used to perform action recognition on a number of feature data using an action recognition model based on a semantic guided neural network to obtain a training action recognition result of the trainee;

[0034] The counting module is used to count the number of corresponding peaks and troughs in a number of characteristic data using a counting algorithm to obtain the number of training movements of the trainee.

[0035] Furthermore, the feature extraction module includes a skeleton point data extraction unit and a feature extraction unit, wherein:

[0036] The skeleton point data extraction unit is used to input each frame of the training video image into the BlazePose network model after image transformation to obtain the skeleton point data of the human body in each frame of the image;

[0037] The feature extraction unit is used to extract features from the human skeleton point data according to the characteristics of the training task to obtain the plurality of feature data, specifically:

[0038] For the rope skipping and squatting movements, the change in the y-axis coordinate of the nose is used as feature f1: f1 = y nose -y start_nose , assuming that the coordinates of the nose at the beginning of the action and the coordinates of the nose at the current moment are (x start_nose ,y start_nose ) and (x nose ,y nose );

[0039] For the squat action, the vertical difference between the knee and the foot is used as feature f2: f2 = y knee -y foot , assuming that the coordinates of the left knee and foot are (x knee ,y knee ) and (x foot ,y foot );

[0040] In the jumping jack movement, the difference between the hand periodically passing above and below the nose is used as feature f3: f3 = y hand -y nose ,y hand Represents the ordinate of the hand, y nose represents the vertical coordinate of the nose;

[0041] During the movement, the angle between the shoulder and the elbow is used as feature f4: Assume that the coordinates of the shoulder and elbow joints are (x shoulder ,y shoulder ) and (x elbow ,y elbow ).

[0042] Furthermore, the action recognition model based on semantically guided neural network includes a joint-level module, a frame-level module and a fully connected layer. The plurality of feature data are dynamically represented as input of the joint-level module, the output of the joint-level module is connected to the fully connected layer via the frame-level module, and the output of the fully connected layer is the training action recognition result of the trainee.

[0043] Compared with the prior art, the present invention has the following technical effects: the present invention takes advantage of the portable characteristics of mobile devices and the excellent performance of the BlazePose network model to run the algorithm in real time on the client, adopts the action recognition model based on semantic guided neural network for training action recognition, and adopts the counting algorithm to calculate the number of peaks and troughs of the data in the feature data to determine the number of times the trainee completes the action. Based on deep learning technology, the present invention only requires a mobile client to realize the recognition and counting of training actions, and has the advantages of low cost, real-time detection, and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0045] Figure 1 It is a flow chart of a method for identifying and counting movements in training without equipment;

[0046] Figure 2 It is a flowchart of a method for recognizing and counting movements of training without equipment based on a mobile terminal;

[0047] Figure 3 It is the bone joint point output by BlazePose;

[0048] Figure 4 It is a diagram of an action recognition model based on a semantically guided neural network;

[0049] Figure 5 It is a schematic diagram of peak and trough detection;

[0050] Figure 6 The present invention is a structural diagram of a motion recognition and counting system for non-equipment training. DETAILED DESCRIPTION

[0051] In order to further illustrate the features of the present invention, please refer to the following detailed description and drawings of the present invention. The drawings are for reference and illustration only and are not intended to limit the scope of protection of the present invention.

[0052] like Figure 1 to Figure 2 As shown, this embodiment discloses a method for identifying and counting non-equipment training movements, comprising the following steps S1 to S4:

[0053] S1, collecting training video images of trainees;

[0054] S2, using the BlazePose network model to process the training video image to obtain some feature data;

[0055] S3, using a semantic guided neural network based action recognition model to perform action recognition on a number of feature data to obtain a training action recognition result of the trainee;

[0056] S4. Use a counting algorithm to count the number of corresponding peaks and troughs in a number of characteristic data to obtain the number of training movements of the trainee.

[0057] It should be noted that this embodiment collects the training video images of the trainee through the mobile terminal, uses the BlazePose network model combined with the mobile phone camera to collect the position information of the human body's skeletal points, extracts features from the skeletal point information, identifies the training movements, and uses a counting algorithm to count the number of times the training movements are completed. This method can be run directly on a mobile phone without relying on other devices. Combined with a non-equipment training method, it can effectively supervise and self-check the trainee's movements.

[0058] As a further preferred technical solution, the above step S2: using the BlazePose network model to process the training video image to obtain several feature data, including:

[0059] Each frame of the training video image is transformed and then input into the BlazePose network model to obtain the skeleton point data of the human body in each frame of the image;

[0060] According to the characteristics of the training task, feature extraction is performed on the human skeleton point data to obtain the plurality of feature data.

[0061] It should be noted that each frame of the training video image is transformed and then input into the BlazePose network model to obtain the human skeleton point data in each frame. The sliding window is used to extract the features of the human skeleton node data. The size of each window is 40 frames of data. A window contains 40*S-dimensional feature data, where S is the number of feature values. The human skeleton node data obtained by the BlazePose network model for processing the training video image is as follows: Figure 3 As shown, the details are as follows:

[0062] (1) Set the window size and step size of the sliding window. The window size is set to 5 frames and the moving step size is 1.

[0063] (2) Assuming that the total number of data frames is N, starting from the i-th frame (2<i<N-2), taking the i-th frame data as the midpoint, select 2 frames of data forward and backward respectively, and a total of 5 frames of data constitute a window.

[0064] (3) Calculate the average value x of the data in the window in step (2) i , x i As the value after filtering the i-th frame data.

[0065] Repeat steps (2) and (3) to continue processing the data of frame i+1 until i reaches the boundary position (i=N-2).

[0066] As a further preferred technical solution, feature extraction is performed on several human skeleton node data according to the characteristics of the training task. The specific features are as follows:

[0067] Feature 1: For the rope skipping and squatting actions, assume that the coordinates of the nose at the beginning of the action and the coordinates of the nose at the current moment are (x start_nose ,y start_nose ) and (x nose ,y nose ). Therefore, the change in the y-axis coordinate of the nose constitutes the feature: f1 = y nose -y start_nose ;

[0068] Feature 2: For squatting, the difference between the vertical values ​​of the knee and foot will show obvious periodic changes, and the change range is large. Therefore, the difference between the vertical values ​​of the knee and foot can be selected as feature f2. Since the left and right joints can be regarded as approximately symmetrical during human movement, only the left half of the body data can be selected when calculating the features. Assume that the coordinates of the left half of the body knee and foot are (x knee ,y knee ) and (x foot ,y foot ), then: f2=y knee -y foot ;

[0069] Feature 3: In the jumping jack exercise, the hand will periodically pass above and below the nose, so the difference is taken as feature f3, then: f3 = y hand -y nose ,y hand Represents the ordinate of the hand, y nose represents the vertical coordinate of the nose;

[0070] Feature 4: During the movement, the angle between the shoulder and the elbow is also an important feature. Assume that the coordinates of the shoulder and elbow joints are (x shoulder ,yshoulder ) and (x elbow ,y elbow ), then:

[0071]

[0072] As a further preferred technical solution, Figure 4 As shown in the figure, the action recognition model based on semantic guided neural network includes joint level module, frame level module and fully connected layer, which learns the dynamic representation of skeletal joints by fusing the position and velocity information of skeletal joints. Two semantics of skeletal joint type and frame index are introduced in the joint level module and frame level module respectively. In order to model the dependency relationship of joints in the joint level module, three GCN layers are used. In order to model the dependency relationship between frames, two CNN layers are used.

[0073] Specifically, the collected image input is transformed into a dynamic representation and then input into the joint-level module. In the joint-level module, C represents the connection, and the dynamic representation is connected with the skeletal joint type. Then, three semantically guided neural networks are used for processing, where X represents matrix multiplication, θ and φ represent two transformation functions, each of which consists of a fully connected layer.

[0074] The output results after processing by the joint-level module are added to the frame sequence. In order to merge all the joint information in a frame, a spatial maximum pooling layer is used to aggregate them across joints. At the same time, two convolutional neural network layers are applied: the first convolutional neural network is a temporal convolutional layer to model the dependencies between frames; the second convolutional neural network enhances the representation ability of the learned features by mapping them to a high-dimensional space with a kernel size of 1. After the two convolutional layers, a temporal maximum pooling layer is used to aggregate the information of all frames, and then a fully connected layer is used to obtain the action recognition results.

[0075] As a further preferred technical solution, the peak and trough counting detection is as follows Figure 5 As shown in the figure, during the training process, each training action of the trainee will have corresponding characteristics that show periodic changes. Therefore, the number of times the trainee has completed the action can be determined based on the number of corresponding peaks and troughs in the characteristic data. The number of peaks and troughs in the characteristic data is calculated to determine the number of times the trainee has completed the action. Only when the peaks and troughs exceed the threshold, a completion is recorded.

[0076] It should be noted that the standing action is used as a transition mark for each action, so the number of times the action is completed does not need to be considered. The training action recognition and counting at least includes: arm extension, jumping jacks, squats, and rope skipping action recognition and counting.

[0077] like Figure 6As shown, this embodiment discloses a non-equipment training motion recognition and counting system, including: an image acquisition module, a feature extraction module, a motion recognition module and a counting module, wherein:

[0078] The image acquisition module is used to acquire the training video images of the trainee;

[0079] The feature extraction module is used to process the training video image using the BlazePose network model to obtain a number of feature data;

[0080] The action recognition module is used to perform action recognition on a number of feature data using an action recognition model based on a semantic guided neural network to obtain a training action recognition result of the trainee;

[0081] The counting module is used to count the number of corresponding peaks and troughs in a number of characteristic data using a counting algorithm to obtain the number of training movements of the trainee.

[0082] As a further preferred technical solution, the feature extraction module includes a skeleton point data extraction unit and a feature extraction unit, wherein:

[0083] The skeleton point data extraction unit is used to input each frame of the training video image into the BlazePose network model after image transformation to obtain the skeleton point data of the human body in each frame of the image;

[0084] The feature extraction unit is used to extract features from the human skeleton point data according to the characteristics of the training task to obtain the plurality of feature data, specifically:

[0085] For the rope skipping and squatting movements, the change in the y-axis coordinate of the nose is used as feature f1: f1 = y nose -y start_nose , assuming that the coordinates of the nose at the beginning of the action and the coordinates of the nose at the current moment are (x start_nose ,y start_nose ) and (x nose ,y nose );

[0086] For the squat action, the vertical difference between the knee and the foot is used as feature f2: f2 = y knee -y foot , assuming that the coordinates of the left knee and foot are (x knew ,y knee ) and (x foot ,y foot );

[0087] In the jumping jack movement, the difference between the hand periodically passing above and below the nose is used as feature f3: f3 = y hand -ynose ,y hand Represents the ordinate of the hand, y nose represents the vertical coordinate of the nose;

[0088] During the movement, the angle between the shoulder and the elbow is used as feature f4: Assume that the coordinates of the shoulder and elbow joints are (x shoulder ,y shoulder ) and (x elbow ,y elbow ).

[0089] As a further preferred technical solution, the action recognition model based on semantically guided neural network includes a joint-level module, a frame-level module and a fully connected layer. The plurality of feature data are dynamically represented and used as input of the joint-level module. The output of the joint-level module is connected to the fully connected layer via the frame-level module. The output of the fully connected layer is the training action recognition result of the trainee.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying and counting training movements without equipment, characterized in that: include: Collecting training video images of trainees; The BlazePose network model is used to process the training video images to obtain some feature data; The action recognition model based on semantic guided neural network is used to perform action recognition on several feature data to obtain the training action recognition results of the trainee; A counting algorithm is used to count the number of corresponding peaks and troughs in a number of characteristic data to obtain the number of training movements of the trainee; The BlazePose network model is used to process the training video image to obtain several feature data, including: Each frame of the training video image is transformed and then input into the BlazePose network model to obtain the skeleton point data of the human body in each frame of the image; Extracting features from human skeleton point data according to the characteristics of the training task to obtain the plurality of feature data; The feature extraction of the human skeleton point data according to the characteristics of the training task to obtain the plurality of feature data includes: For the rope skipping and squatting actions, the change in the y-axis coordinate of the nose is used as the feature f1: f1 = y nose -y start_nose , assuming that the coordinates of the nose at the beginning of the action and the coordinates of the nose at the current moment are (x start_nose ,y start_nose ) and (x nose ,y nose ); For the squat action, the vertical difference between the knee and the foot is used as feature f2: f2 = y knee -y foot , assuming that the coordinates of the left knee and foot are (x knee ,y knee ) and (x foot ,y foot ); In the jumping jack movement, the difference between the hand periodically passing above and below the nose is used as feature f3: f3 = y hand -y nose ,y hand Represents the ordinate of the hand, y nose represents the vertical coordinate of the nose; During the movement, the angle between the shoulder and the elbow is used as feature f4: Assume that the coordinates of the shoulder and elbow joints are (x shoulder ,y shoulder ) and (x elbow ,y elbow ); The action recognition model based on the semantic guidance neural network includes a joint level module, a frame level module and a fully connected layer. The action recognition model based on the semantic guidance neural network is used to perform action recognition on a number of feature data to obtain a training action recognition result of the trainee, including: transforming the plurality of feature data into dynamic representations and inputting them into a joint-level module; The joint-level module learns the dynamic representation of skeletal joints by fusing the position and velocity information of skeletal joints and models the dependencies of joints. The frame-level module aggregates the output of the joint-level module to obtain all joint information of each frame, and models the dependency between frames to obtain joint information of all frames. The fully connected layer obtains the trainee's action based on the output recognition of the frame-level module.

2. The method for identifying and counting non-equipment training movements according to claim 1, characterized in that: The joint-level module learns the dynamic representation of skeletal joints by fusing the position and velocity information of skeletal joints and models the dependency of joints, including: Connecting the dynamic representations of the plurality of feature data with the bone joint point types to obtain joint data; A semantically guided neural network is used to process joint data and model the dependencies between joints.

3. The method for identifying and counting non-equipment training movements according to claim 1, characterized in that: The frame-level module aggregates the output of the joint-level module to obtain all joint information of each frame, and models the dependency between frames to obtain joint information of all frame images, including: The output of the joint level module is added to the frame sequence, and all joint information in each frame is aggregated using a spatial maximum pooling layer to obtain an aggregated result; A convolutional neural network is used to model the dependencies between frames.

4. The method for identifying and counting non-equipment training movements according to claim 1, characterized in that: The identified and counted training movements include at least arm extension, jumping jacks, squats, and rope skipping.

5. A non-equipment training action recognition and counting system, characterized in that: include: Image acquisition module, feature extraction module, action recognition module and counting module, where: The image acquisition module is used to acquire the training video images of the trainee; The feature extraction module is used to process the training video image using the BlazePose network model to obtain a number of feature data; The action recognition module is used to perform action recognition on a number of feature data using an action recognition model based on a semantic guided neural network to obtain a training action recognition result of the trainee; The counting module is used to use a counting algorithm to count the number of corresponding peaks and troughs in a number of characteristic data to obtain the number of training movements of the trainee; The feature extraction module includes a skeleton point data extraction unit and a feature extraction unit, wherein: The skeleton point data extraction unit is used to input each frame of the training video image into the BlazePose network model after image transformation to obtain the skeleton point data of the human body in each frame of the image; The feature extraction unit is used to extract features from the human skeleton point data according to the characteristics of the training task to obtain the plurality of feature data, specifically: For the rope skipping and squatting movements, the change in the y-axis coordinate of the nose is used as feature f1: f1 = y nose -y start_nose , assuming that the coordinates of the nose at the beginning of the action and the coordinates of the nose at the current moment are (x start_nose ,y start_nose ) and (x nose ,y nose ); For the squat action, the vertical difference between the knee and the foot is used as feature f2: f2 = y knee -y foot , assuming that the coordinates of the left half of the body's knee and foot are (x knee ,y knee ) and (x foot ,y foot ); In the jumping jack movement, the difference between the hand periodically passing above and below the nose is used as feature f3: f3 = y hand -y nose ,y hand Represents the ordinate of the hand, y nose represents the vertical coordinate of the nose; During the movement, the angle between the shoulder and the elbow is used as feature f4: Assume that the coordinates of the shoulder and elbow joints are (x shoulder ,y shoulder ) and (x elbow ,y elbow ); The action recognition model based on semantically guided neural network includes a joint-level module, a frame-level module and a fully connected layer. The plurality of feature data are dynamically represented as input to the joint-level module, the output of the joint-level module is connected to the fully connected layer via the frame-level module, and the output of the fully connected layer is the training action recognition result of the trainee.

Images