A motion posture correction method based on action recognition

Abstract

The present invention relates to a motion posture correction method based on motion recognition, and belongs to the technical field of computer and motion recognition. The method comprises: motion recognition: collecting user motion videos through a camera, constructing a motion recognition algorithm based on a deep learning algorithm, and using the algorithm to classify the motions in the exerciser's video; skeleton extraction: first extracting the standard motions of the corresponding exerciser from a standard fitness database, and then extracting the skeleton from the standard motion video at the same time; at the same time, using the collected exerciser video, extracting the exerciser's motion skeleton model from it; posture comparison: firstly standardizing the exerciser's skeleton template; then calculating the similarity between the standardized exerciser's skeleton and the standard motion video, extracting a frame of template motion with the highest motion similarity; finally, all motion tests are completed, selecting several frames of motion with similarity lower than a threshold, and displaying the template motion. The present invention improves the calculation speed and comparison accuracy.

Description

Technical Field

[0001] The invention belongs to the technical field of computers and motion recognition, and relates to a motion posture correction method based on motion recognition. Background Art

[0002] Fitness apps are a type of fitness resource that can provide the general public with professional fitness plans, standard movement videos, and other fitness resources. However, fitness apps can only provide reference standard movements, and cannot provide users with detailed guidance on movement implementation like a fitness coach. Therefore, in many cases during actual exercise, users may suffer injuries due to non-standard movements caused by simply imitating standard movements.

[0003] At present, commonly used human behavior detection methods include electromyography detection, airbag sensor information acquisition, visual image methods, etc.

[0004] The method based on electromyography detection uses the biological electromyography signals generated by human movement to identify human movements, but the user needs to wear sensors when using it. It is mostly used for scientific research in specific scenarios and does not meet the needs of daily fitness.

[0005] The airbag-based method is similar to the electromyography detection method, and also requires the user to wear relevant sensors during exercise to obtain their movement information. Therefore, whether it is the electromyography detection method or the airbag-based method based on the use of wearable sensors, the user needs to wear sensors, which is not suitable for posture correction in daily fitness exercises.

[0006] Methods based on visual images. This type of method requires using images taken by the user's device to estimate the user's posture and action, including user outline estimation and user skeleton estimation. The main application is Openpose, which uses a large amount of human activity data and labels to train a graph neural network to recognize human posture. However, its disadvantage is that it has high performance for the device, and the real-time performance of this algorithm is poor in general devices. It is generally used for posture recognition applications in single-frame images in experiments, and cannot complete the estimation and comparison of user posture.

[0007] Therefore, there is an urgent need for a more convenient and accurate fitness auxiliary system to simulate and compare fitness by oneself. Summary of the invention

[0008] In view of this, the purpose of the present invention is to provide a motion posture correction method based on motion recognition to solve the problems of lack of professional guidance in daily individual exercise, or too complicated professional equipment and too large algorithm. First, the motion recognition classification and skeleton extraction of the exerciser's action video are carried out to obtain the skeleton material of the exerciser's own action video and the corresponding standard action video. Then, a posture comparison algorithm based on video key frame positioning is designed to achieve high-precision comparison between users and standard actions, and provide exercisers with professional guidance that is difficult to obtain in fragmented time exercise. Finally, according to the movement of the crowd, the corresponding motion posture recognition data set is produced.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A motion posture correction method based on action recognition specifically comprises the following steps:

[0011] S1: Action recognition stage:

[0012] The user's sports video is collected through the user's PC or mobile camera, and a sufficiently accurate action recognition algorithm is built based on the deep learning algorithm. The algorithm is used to classify the actions in the exerciser's video;

[0013] S2: Skeleton extraction stage:

[0014] First, the standard movements of the corresponding exercisers are extracted from the standard fitness database, and then the skeleton is extracted from the standard movement video at the same time; at the same time, the skeleton model of the exerciser's movement is extracted from the collected exerciser video;

[0015] S3: Posture comparison stage:

[0016] First, the skeleton of the exerciser is standardized as a template; then, the similarity between the standardized skeleton of the exerciser and the standard action video is calculated to extract the frame of template action with the highest action similarity; finally, after all the action tests of the exerciser are completed, several frames of action with similarity below the threshold are selected and the template action is displayed.

[0017] Furthermore, in step S1, in the action recognition stage, a C3D network is specifically used for action recognition.

[0018] Furthermore, in step S1, the C3D network includes: 8 convolution operations, 5 pooling operations, and finally 2 fully connected layers and softmax output to obtain the result.

[0019] Further, in step S2, the skeleton extraction specifically includes: firstly, using the Faster R-CNN network and the Feature Pyramid Networks (FPN) to perform detection, and according to the detection results, extracting the target occupying the largest area in the image, and defaulting it to the exerciser; and then using the High resolution network (HRNet) to estimate the human body posture.

[0020] Furthermore, in step S2, the Faster R-CNN network (Full convolution depth network) includes: a full convolution depth network (Region Proposal Network, RPN) and a Fast R-CNN detector; the specific algorithm is: obtaining a feature map through a convolutional neural network, inputting the feature map into the RPN network to obtain a candidate region, and inputting the candidate region and the feature map into the full convolution layer for convolution classification, and finally outputting the result.

[0021] Further, in step S2, the FPN network includes: 1) a bottom-up network forward propagation process; 2) a top-down process of fusing top-level features with bottom-level features through upsampling, and each level is predicted independently.

[0022] Furthermore, in step S2, the fusion of Faster R-CNN and FPN is mainly performed bottom-up and top-down fusion on the RPN part to obtain a more appropriate ROI area.

[0023] Further, in step S2, the internal structure of the HR network includes: starting with a high-resolution subnet as the first stage, adding subnets from high to low resolution one by one to form more stages, and connecting multi-resolution subnets in parallel. In the whole process, the information in the parallel multi-resolution subnets is repeatedly exchanged to perform repeated multi-scale fusion.

[0024] Further, in step S3, the template standardization specifically includes: discarding the four positions of eyes and ears obtained by posture estimation, retaining the positions of 13 joint points such as nose, shoulder, elbow, etc., and recording the joint point as J, which is expressed as J = (j 0 、j 1 、j 2 ,...,j 12 ), each joint point contains two dimensions x and y; the template action information is recorded as The action information to be calculated is recorded as

[0025] Calculate the skeleton lengths of the template action t and the action to be calculated a according to the joint points, and then transform the skeleton length of a to the corresponding skeleton length of t; after the skeleton length of a is transformed, select the reference points of t and a, and translate a to the target position in a way that the reference points coincide;

[0026] Specifically, according to the relationship of the human body structure, the joint positions of the hands are the wrists, elbows, and shoulders, and the key positions of the feet are the ankles, knees, and hips. The nose is used as a fixed point to consider the positions of the nose and shoulders, and the positions of the shoulders and hips. The starting point is s and the end point is e.

[0027] The distance between two joint positions is calculated as:

[0028]

[0029] Then calculate the length that needs to be transformed between joint points:

[0030]

[0031]

[0032]

[0033] Let the transformation length be M;

[0034] Then take the nose as the fixed point, that is, the key point 0 does not move. Taking the left wrist as an example, the transformed coordinates are

[0035]

[0036] Finally, the template nose is used as the reference point to translate the estimated image.

[0037] Further, in step S3, calculating the similarity specifically includes: matching the joint points of the template with the joint points of the test video one by one, calculating the distance between the two, and summing the distances between the points to D; therefore, the smaller the distance, the more similar the two actions are. Following the Sigmoid function, ε is a preset error factor, and the relationship between the distance and the similarity S is:

[0038]

[0039] The test video is calculated by taking frames at intervals of fps / 5. The obtained image is calculated similar to the template image. The template frame with the highest similarity is assumed to be the same action as the person in the image at that time. Finally, the similarity of all frames extracted from the entire video is obtained. Finally, a similarity threshold is set manually. When the image is lower than this threshold, it is considered to be an incompatible action. The standard action and the incompatible action at that time are output, and the action is adjusted through manual comparison.

[0040] The beneficial effect of the present invention is that the present invention combines the three aspects of target detection, posture recognition, and abnormal judgment into one aspect, which avoids the need for sensors in traditional methods. At the same time, it uses the images obtained by users when using apps such as "KEPP" and standard images to intelligently compare and analyze the user's movement status and correct the user's movement pattern. On this basis, users can get more accurate exercise guidance, and can use the infrared, RFID and other technologies built into the mobile phone to further understand their physical condition, obtain information including temperature, heart rate, and physical condition of various parts, to achieve better exercise results. The specific advantages are as follows:

[0041] 1) In terms of functions, the present invention has multiple functions such as human body detection, motion posture acquisition, posture comparison, anomaly detection and intelligent adjustment, which cannot be accomplished by any single application at present.

[0042] 2) In terms of performance, the algorithm of the present invention avoids the requirements of methods such as OpenPose on computer performance, and realizes real-time and fast human posture and motion detection.

[0043] 3) In terms of algorithms, the real-time human posture detection algorithm proposed in the present invention has a high level of performance and accuracy, taking into account both real-time and accuracy, and is suitable for integration in different applications; at the same time, a motion posture comparison algorithm based on key frame positioning is proposed to further improve the comparison accuracy.

[0044] The motion posture detection and comparison algorithm included in the present invention realizes the intelligent inference of human body state by using multi-mode information and provides the calibration information required by the user. At the same time, the expert system and fuzzy reasoning are used to combine various information of the user to macro-control the further exercise direction and exercise content of the user.

[0045] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 This is a flow chart of the dynamic correction method based on motion recognition of the present invention;

[0048] Figure 2 This is the C3D network structure diagram;

[0049] Figure 3 This is the Faster R-CNN network structure diagram;

[0050] Figure 4 It is the structure diagram of FPN;

[0051] Figure 5 This is the HR network structure diagram;

[0052] Figure 6 is the pose estimation graph;

[0053] Figure 7 For the template diagram, Figure 7 (a) is the original template image. Figure 7 (b) is the joint point of the human body template without background;

[0054] Figure 8 This is the posture comparison process diagram. Figure 8 (a) is the original test image. Figure 8 (b) is a node without background exerciser, Figure 8 (c) is posture alignment, Figure 8 (d) is the image after translation. DETAILED DESCRIPTION

[0055] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0056] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0057] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0058] See also Figures 1 to 8 The present invention designs a motion posture correction method based on action recognition, which specifically includes the following steps:

[0059] S1: Action recognition stage: Collect user motion videos through the camera on the user's PC or mobile device, build an action recognition algorithm with sufficient accuracy based on the deep learning algorithm, and use the algorithm to categorize the actions in the exerciser's video.

[0060] Preferably, in the action recognition stage, a C3D network can be used for action recognition. The C3D network takes the video as input, retains the input time information, and outputs a 3D feature map. The C3D network includes: 8 convolution operations, 5 pooling operations, and finally obtains the result through 2 full connection layers and softmax output.

[0061] S2: Skeleton extraction stage: First, the standard movements of the corresponding exercisers are extracted from the standard fitness database, and then the skeleton is extracted from the standard movement video at the same time; at the same time, the skeleton model of the exerciser's movement is extracted from the collected exerciser video.

[0062] Preferably, firstly, Faster R-CNN network (second position) and FPN network (third position) are used for detection, and according to the detection results, the target with the largest area in the image is extracted and regarded as the exerciser by default. Finally, the human posture estimation network HR network (fourth position) is used to estimate the human posture.

[0063] (1) Exerciser detection:

[0064] The Faster R-CNN network is mainly composed of two modules: Figure 3As shown in the figure, the first module is a fully convolutional deep network (RPN) based on candidate regions, and the second module is a Fast R-CNN detector. The RPN module can provide candidate regions to the detector, and the detector then identifies objects. The overall algorithm is to obtain feature maps through convolutional neural networks, input feature maps into the RPN network to obtain candidate regions, and input candidate regions and feature maps into the full convolution layer for convolution classification, and finally output the results.

[0065] The FPN network consists of two parts, such as Figure 4 As shown in the figure, in the bottom-up network forward propagation process, in this part, the size of the feature map will gradually decrease, and high-level features will be obtained. The top-down process is to fuse the top-level features with the bottom-level features through upsampling, and each level is predicted independently.

[0066] The fusion of Faster R-CNN and FPN mainly performs bottom-up and top-down fusion on the RPN part to obtain a more suitable ROI area.

[0067] (2) Human Pose Estimation

[0068] The high-resolution network (HRNet) structure is as follows Figure 5 As shown in the figure, the main feature of the network is that the feature map always maintains high-resolution representation throughout the process, and uses repeated multi-resolution representation information exchange to enhance its expressiveness, so that the representation learned by the model has a significant improvement in spatial accuracy. The model gradually joins the low-resolution feature map sub-network in parallel through the high-resolution feature map main network to achieve multi-scale fusion and feature extraction of different networks. The biggest difference between the HR network and the previous network is that the resolutions are connected in parallel.

[0069] S3: Posture comparison stage: Since the proportions of the human body are inconsistent, such as adults and children, women and men, the elderly and the young, etc. In order to make the algorithm adapt to different people at any stage, before calculating the similarity, the skeleton of the exerciser is firstly standardized; then, the similarity between the standardized skeleton of the exerciser and the standard action video is calculated, and the frame of template action with the highest action similarity is extracted; finally, after all the action tests of the exerciser are completed, several frames of action with similarity below the threshold are selected, and the template action is displayed.

[0070] (1) Standardization

[0071] Discard the four positions of eyes and ears obtained by posture estimation, and retain the positions of 13 joint points such as nose, shoulders, elbows, etc. Figure 6 As shown, the joint point is recorded as J, expressed as J = (j 0 、j 1 、j 2 ,...,j12 ), each joint point contains two dimensions x and y; the template action information is recorded as The action information to be calculated is recorded as

[0072] The skeleton lengths of the template action t and the action to be calculated a are calculated according to the joint points, and then the skeleton length of a is transformed to the corresponding skeleton length of t; after the skeleton length of a is transformed, the reference points of t and a are selected, and a is translated to the target position in a way that the reference points coincide.

[0073] The specific method is to follow the relationship between the human body structure, the joint positions of the hands are wrists, elbows, and shoulders, the key positions of the feet are ankles, knees, and hips, and take the nose as the fixed point, consider the position of the nose and shoulders, and the position of the shoulders and hips; remember the starting point as s and the end point as e.

[0074] Table 1 Attitude start-end table

[0075] s e 0-> 1-> 2-> 3 0-> 4-> 5-> 6 1-> 7-> 8-> 9 4-> 10-> 11-> 12

[0076] According to the sorting of Table 1 above, the distance between the joint positions is calculated as follows:

[0077]

[0078] Then calculate the length that needs to be transformed between joint points:

[0079]

[0080]

[0081]

[0082] Let the transformation length be M.

[0083] Then take the nose as the fixed point, that is, the key point 0 does not move. Taking the left wrist as an example, the transformed coordinates are

[0084]

[0085] Finally, the template nose is used as the reference point to translate the estimated image.

[0086] (2) Similarity calculation and results

[0087] The joint points of the template are matched with the joint points of the test video one by one, the distance between the two is calculated, and the distance between each point is summed up to D. Therefore, the smaller the distance, the more similar the two actions are. Following the Sigmoid function, ε is the preset error factor, and the relationship between the distance and the similarity S is:

[0088]

[0089] The test video is calculated by taking frames at intervals of fps / 5. Figures 7-8 As shown in the figure, the obtained image is similar to the template image. The template frame with the highest similarity is assumed to be the one that performs the same action as the person in the image at this time. Finally, the similarity of all frames extracted from the entire video is obtained. Finally, a similarity threshold is set manually. When the image is lower than this threshold, it is considered to be an incompatible action. The standard action and the incompatible action at this time are output, and the action is adjusted through manual comparison.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A motion posture correction method based on motion recognition, characterized in that: The method specifically comprises the following steps: S1: Action recognition stage: The user's exercise videos are collected through the camera, and an action recognition algorithm is built based on the deep learning algorithm. The algorithm is used to classify the actions in the exerciser's video. S2: Skeleton extraction stage: First, the standard movements of the corresponding exercisers are extracted from the standard fitness database, and then the skeleton is extracted from the standard movement video at the same time; at the same time, the skeleton model of the exerciser's movement is extracted from the collected exerciser video; The skeleton extraction specifically includes: firstly, using the Faster R-CNN network and the Feature Pyramid Networks (FPN) to perform detection, and according to the detection results, extracting the target with the largest area in the image, and defaulting it to the exerciser; then using the High Resolution Network (HRNet) to estimate the human body posture; The Faster R-CNN network (Full convolution depth network) includes: a full convolution depth network (Region Proposal Network, RPN) and a Fast R-CNN detector; the specific algorithm is: obtain a feature map through a convolutional neural network, input the feature map into the RPN network to obtain a candidate region, and input the candidate region and the feature map into the full convolution layer for convolution classification, and finally output the result; The FPN network includes: 1) a bottom-up network forward propagation process; 2) a top-down process of fusing top-level features with bottom-level features through upsampling, with each level independently predicted; The fusion of Faster R-CNN and FPN is to perform bottom-up and top-down fusion on the RPN part to obtain a more appropriate ROI area; The internal structure of the HR network is as follows: starting with a high-resolution subnet as the first stage, adding subnets from high to low resolution one by one to form more stages, and connecting multi-resolution subnets in parallel; repeatedly exchanging information in parallel multi-resolution subnets throughout the process for repeated multi-scale fusion; S3: Posture comparison stage: First, the skeleton of the exerciser is standardized as a template; then, the similarity between the standardized skeleton of the exerciser and the standard action video is calculated to extract the frame of template action with the highest action similarity; finally, after all the action tests of the exerciser are completed, several frames of action with similarity below the threshold are selected and the template action is displayed.

2. The method for correcting sports posture according to claim 1, characterized in that: In step S1, in the action recognition stage, a C3D network is specifically used for action recognition.

3. The method for correcting sports posture according to claim 2, characterized in that: In step S1, the C3D network includes: 8 convolution operations, 5 pooling operations, and finally 2 fully connected layers and softmax output to obtain the result.

4. The method for correcting sports posture according to claim 1, characterized in that: In step S3, template standardization specifically includes: discarding the four positions of eyes and ears obtained by posture estimation, retaining the positions of 13 joint points, and recording the joint points as J, which is expressed as J = (j0, j1, j2, ..., j 12 ), each joint point contains two dimensions x and y; the template action information is recorded as The action information to be calculated is recorded as Calculate the skeleton lengths of the template action t and the action to be calculated a according to the joint points, and then transform the skeleton length of a to the corresponding skeleton length of t; after the skeleton length of a is transformed, select the reference points of t and a, and translate a to the target position in a way that the reference points coincide; Specifically, according to the relationship of the human body structure, the joint positions of the hands are the wrists, elbows, and shoulders, and the key positions of the feet are the ankles, knees, and hips. The nose is used as a fixed point to consider the positions of the nose and shoulders, and the positions of the shoulders and hips. The starting point is s and the end point is e. The distance between two joint positions is calculated as: Then calculate the length that needs to be transformed between joint points: Let the transformation length be M; Then take the nose as the fixed point, that is, the key point 0 does not move. Taking the left wrist as an example, the transformed coordinates are Finally, the template nose is used as the reference point to translate the estimated image.

5. The method for correcting sports posture according to claim 4, characterized in that: In step S3, calculating the similarity specifically includes: matching the joint points of the template with the joint points of the test video one by one, calculating the distance between the two, and summing the distances between the points to D; following the Sigmoid function, ε is a preset error factor, and the relationship between the distance and the similarity S is:

Images