A fitness action data acquisition and analysis method based on a video deep learning algorithm

Abstract

The present application relates to the technical field of video image data analysis, and particularly relates to a bodybuilding motion data acquisition and analysis method based on a video deep learning algorithm. Continuous motion image frames of a subject to be measured are acquired to obtain human skeleton joint nodes, joint edges and feature point motion trajectory sequences. For image frames with missing joint nodes, feature points on joint edges to which the missing joint nodes belong are used as reference points, and a position change difference feature is combined to calculate an effective factor and a position fitting smoothness to determine a smoothing factor. The effective factor, the smoothing factor and the spatial distribution law of the reference points are fused to make the predicted position of the missing joint node conform to the linear constraint of the human joint edge. Meanwhile, the joint nodes and the feature points are associated in the first image to determine the matching motion trajectory sequences. By analyzing the change consistency of the matching motion trajectory sequences of the missing joint nodes and other joint nodes on the basis of the predicted position, the reliability of the final position is ensured, and the accuracy of the motion data acquisition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video image data analysis, and particularly relates to a bodybuilding action data acquisition and analysis method based on a video deep learning algorithm. BACKGROUND

[0002] With the improvement of the national health consciousness and the rapid development of the bodybuilding industry, the scientific, personalized and family-oriented bodybuilding demand is increasing. By relying on AI visual recognition, Internet of Things, Internet, cloud computing, big data analysis and other technologies, a system integrating human body recognition, exercise data acquisition, exercise action recognition, exercise action quality recognition, voice broadcast, exercise data statistics and analysis and other functions can be established to realize the intelligent acquisition, statistics and analysis of the exercise data of traditional bodybuilding equipment, and improve the exercise experience of the exercisers and the intelligent level of the traditional bodybuilding equipment.

[0003] In the prior art, when collecting bodybuilding action data, the position and posture of the human body in the image are usually tracked, for example, the motion change characteristics of the human body joint nodes are tracked, so as to realize the recognition and analysis of the bodybuilding action. However, in the actual motion scene, due to the change of the position and posture of the human body, the joint nodes may be missing in some image frames due to occlusion, which may cause the motion change characteristics of the joint nodes to be discontinuous. If the position of the missing joint nodes is predicted only according to a simple adjacent frame interpolation method, the repaired position may not match the actual action, and does not conform to the human body kinematics law, which seriously affects the accuracy of the motion data acquisition. SUMMARY

[0004] In order to solve the technical problem that due to the change of the position and posture of the human body, the joint nodes may be missing in some image frames due to occlusion, which may cause the motion change characteristics of the joint nodes to be discontinuous, and if the position of the missing joint nodes is predicted only according to a simple adjacent frame interpolation method, the repaired position may not match the actual action, and does not conform to the human body kinematics law, which seriously affects the accuracy of the motion data acquisition, the purpose of the present application is to provide a bodybuilding action data acquisition and analysis method based on a video deep learning algorithm, and the technical scheme adopted is as follows:

[0005] In the continuous motion image frames of the to-be-tested person, the human body skeleton detection result in each motion image frame and the motion trajectory sequence of each feature point are acquired, wherein the human body skeleton detection result includes joint edges and joint nodes;

[0006] In the motion image frame in which the joint node is missing, all feature points on the joint edge to which the missing joint node belongs are taken as reference points; for each missing joint node, the position change difference features of the reference points in the motion trajectory sequence are analyzed, the effective factors of each reference point are determined; the position fitting smoothness between the reference points is analyzed, the smoothness factors of each reference point on the joint edge are determined; based on the effective factors, the smoothness factors of the reference points corresponding to each missing joint node and the position distribution of the reference points, the predicted position of each missing joint node is obtained;

[0007] In the first motion image frame, based on the position relationship between the joint nodes and the feature points, the matching motion trajectory sequence of each joint node is determined; based on the predicted position of the missing joint node, the change consistency degree of the matching motion trajectory sequence between the missing joint node and other joint nodes is analyzed, so that the final position of the missing joint node is determined.

[0008] Further, the method for obtaining the motion trajectory sequence of each feature point comprises:

[0009] Based on a target detection algorithm, a human body region is framed in each motion image frame, and in the human body region, an angle point is obtained as a feature point based on an angle point detection algorithm;

[0010] In all motion image frames, the position of each feature point in the first motion image frame is tracked based on an optical flow method, so that the motion trajectory sequence of each feature point is obtained, and the points on the motion trajectory sequence are all taken as feature points.

[0011] Further, the method for obtaining the effective factor comprises:

[0012] The motion image frame in which the joint node is missing is taken as a target frame;

[0013] In each target frame, the position change of each reference point of the missing joint node in the motion trajectory sequence is analyzed, the motion vector of the reference point in each motion image frame is obtained, so that two features of each reference point are determined, the two features are respectively a module length and an angle value;

[0014] For any one feature of each reference point, the standard deviation of all values of each reference point in the feature is taken as a first invalid parameter of each reference point, the absolute value of the difference between the value of each reference point in the target frame in the feature and the average value of the value in the motion trajectory sequence in the feature is taken as a second invalid parameter of each reference point, and the value obtained by negatively correlating and normalizing the product of the first invalid parameter and the second invalid parameter of each reference point is taken as the effective factor corresponding to the feature of each reference point.

[0015] Further, in each target frame, the position change in the motion trajectory sequence of each reference point missing the joint node is analyzed to obtain the motion vector of the reference point under each motion image frame, so as to determine two features of each reference point, including:

[0016] In each target frame, in the motion trajectory sequence of each reference point missing the joint node, each coordinate position is taken as an end point, and the adjacent previous coordinate position is taken as a start point, so as to obtain the motion vector of the reference point under each motion image, the module length of the motion vector is taken as the first feature of the reference point, and the angle value between the motion vector of the reference point under each motion image and the preset direction is taken as the second feature of the reference point.

[0017] Further, the method for obtaining the smoothing factor includes:

[0018] For any one feature of the reference point, the values of all reference points of each missing joint node under the feature are curve-fitted according to the arrangement order on the joint edge to obtain a first fitting curve.

[0019] The mean square error between the first fitting curve and the values of all reference points under the feature is calculated, and is negatively correlated and normalized to be taken as the smoothing factor of each reference point corresponding to each missing joint node.

[0020] Further, the method for obtaining the predicted position includes:

[0021] In each target frame, an optional non-missing joint node on the joint edge to which the missing joint node belongs is taken as a reference point, and the normalized value of the Euclidean distance between each reference point and the reference point is taken as a distance parameter.

[0022] The effective factor and the smoothing factor of the reference point under each feature are combined to obtain the confidence of the reference point under each feature.

[0023] Under each feature of the reference point, the distance parameter corresponding to the reference point is taken as an abscissa, the value of the reference point under each feature is taken as an ordinate, and the confidence of the reference point under each feature is taken as a weight, so as to perform curve fitting to obtain a second fitting curve of each feature.

[0024] In other motion image frames before the time sequence of the target frame, the motion image frame in which the missing joint node and the reference point in the target frame are both not missing is taken as a comparison frame, in the comparison frame closest to the time sequence of the target frame, a joint node of the same type as the missing joint node is taken as an analysis point, and the normalized value of the Euclidean distance between the analysis point and the joint node of the same type as the reference point is mapped in the second fitting curve under each feature, so as to obtain the value corresponding to the missing joint node in the target frame under each feature.

[0025] According to the length of the motion vector corresponding to the missing joint in the target frame, the angle value and the position coordinate of the analysis point, the predicted position of the missing joint in the target frame is calculated.

[0026] Further, the confidence acquisition method comprises:

[0027] Under each feature of the reference point, the normalized value of the product of the effective factor and the smoothing factor of each reference point is taken as the confidence of each reference point under each feature.

[0028] Further, the final position acquisition method comprises:

[0029] The predicted position of the missing joint is used to replace the coordinates of the feature points belonging to the target frame in the corresponding matching motion trajectory, so as to obtain an updated motion trajectory, and the horizontal and vertical coordinates of all coordinates in the updated motion trajectory are sequentially subjected to Fourier transformation, so as to obtain the periods corresponding to the main peak frequencies of the horizontal and vertical coordinates, respectively.

[0030] The horizontal and vertical coordinates of all feature points in the matching motion trajectory of the reference point corresponding to the missing joint are sequentially subjected to Fourier transformation, so as to obtain the periods corresponding to the main peak frequencies of the horizontal and vertical coordinates, respectively.

[0031] The absolute value of the difference between the period corresponding to the main peak frequency of the horizontal coordinate of the missing joint and the period corresponding to the main peak frequency of the horizontal coordinate of the corresponding reference point is taken as a first difference factor, and the absolute value of the difference between the period corresponding to the main peak frequency of the vertical coordinate of the missing joint and the period corresponding to the main peak frequency of the vertical coordinate of the corresponding reference point is taken as a second difference factor.

[0032] If the first difference factor and the second difference factor corresponding to the missing joint are both less than a preset difference threshold, the predicted position of the missing joint is taken as the final position, otherwise, the median value of the coordinates of the feature points belonging to the target frame and the coordinates of the predicted position in the matching motion trajectory sequence of the missing joint is taken as the final position of the missing joint.

[0033] Further, the matching motion trajectory sequence acquisition method comprises:

[0034] In the first motion image frame, the Euclidean distance between each joint and each feature point is calculated as a distance factor between each joint and each feature point.

[0035] For any joint, the motion trajectory sequence of the feature point with the smallest distance factor and the existing motion trajectory sequence in all distance factors corresponding to the joint is taken as the matching motion trajectory sequence of the joint.

[0036] Further, the difference threshold is 0.5.

[0037] The present application has the following beneficial effects:

[0038] The continuous motion image frames of the to-be-tested person are collected, the joint nodes and joint edges of the human body skeleton in each motion image frame are obtained, and the motion trajectory sequence of the feature points in the motion image frame is tracked, so that the motion change characteristics of the joint nodes can be compensated and analyzed by associating the feature points when the joint nodes are missing in the subsequent process. First, in the motion image frame where the joint nodes are missing, all feature points on the joint edge to which the missing joint nodes belong are used as reference points, and the effective factor is calculated by combining the position change difference characteristics of the reference points in the motion trajectory sequence, and the smoothing factor is determined by the position fitting smoothness, and then the effective factor, the smoothing factor and the spatial distribution law of the reference points are fused, so that the predicted position of the missing joint node is more consistent with the linear constraint of the human joint edge, and the limb distortion error that may be caused by the traditional interpolation method is eliminated. Finally, in the first motion image frame, the joint nodes and the feature points are associated based on the positional relationship between the joint nodes and the feature points, so as to determine the matching motion trajectory sequence of each joint node, and then based on the predicted position of the missing joint node, the consistency degree of the matching motion trajectory sequence of the missing joint node and the adjacent joint is analyzed, so as to ensure that the final position of the missing joint node is more consistent with the human kinematics law, improve the physical rationality of the motion change characteristics, and ensure the accuracy of the motion data acquisition. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 The method flowchart of the bodybuilding motion data acquisition and analysis method based on the video deep learning algorithm provided by one embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following will combine the drawings and the preferred embodiments to specifically describe the specific implementation, structure, features and effects of the bodybuilding motion data acquisition and analysis method based on the video deep learning algorithm according to the present application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0043] The application provides a body exercise data acquisition and analysis method based on a video deep learning algorithm.

[0044] Referring to Figure 1 The method comprises the following steps:

[0045] Step S1: In the continuous motion image frames of the to-be-tested person, the human body skeleton detection result in each motion image frame and the motion trajectory sequence of each feature point are obtained, wherein the human body skeleton detection result comprises joint edges and joint nodes.

[0046] First, the continuous motion image frames (for example, 300 frames of deep squat action sequences, containing multiple motion periods) of the to-be-tested person performing the body exercise are collected by using a monocular RGB camera, and each motion image frame is subjected to image preprocessing, including image denoising, size normalization and the like, so as to improve the stability of subsequent image analysis.

[0047] Then, human body skeleton detection is performed on each motion image frame. Specifically, a target detection algorithm (such as YOLO) can be used to locate the human body bounding box in the motion image frame, and then a key point detection model (such as HRNet) is used to predict the two-dimensional pixel coordinates of multiple key points (such as 17 COCO joint nodes or 25 OpenPose joint nodes) of the human body in each bounding box; thereby the human body skeleton detection result in each motion image frame is output, which comprises joint edges and joint nodes, and the joint nodes are represented in the form of coordinates (x, y), wherein x and y represent pixel coordinates (in order to eliminate the influence of the position and scale of the person in the image, the coordinates of the joint nodes are normalized, for example, a coordinate system can be constructed with the center of the hip joint as the origin, the horizontal right direction and the vertical upward direction as the coordinate axes, and the coordinates are normalized to the [-1, 1] interval), and the score index of each joint node is also obtained, which is a confidence score between 0 and 1, indicating the credibility of the model in obtaining the position of the joint node. When the score index is very low (such as <0.2), it usually means that the joint node is occluded or not detected in the motion image frame, which is regarded as a missing joint node.

[0048] When a person performs a fitness action, some joints may be blocked by the equipment or other parts of the body, resulting in missing joint detection or severe coordinate jitter in some motion image frames, which seriously affects the accuracy of subsequent action type recognition. Therefore, in the embodiment of the present application, the feature points in each motion image frame and the motion trajectory sequence of the feature points are obtained, and the joints and the feature points are associated and analyzed in the subsequent process, so that the missing joints are position predicted based on the motion trajectory sequence of the feature points.

[0049] Preferably, the method for obtaining the motion trajectory sequence of the feature points in the embodiment of the present application comprises:

[0050] Similarly, the human body region is framed in each motion image frame based on a target detection algorithm (such as YOLO), and then the corner points (such as Shi-Tomasi corner points) are obtained as feature points in the human body region based on a corner detection algorithm.

[0051] In all motion image frames, the positions of each feature point in the first motion image frame are tracked based on an optical flow method, so that the motion trajectory sequence of each feature point in the first motion image frame in the entire continuous motion image frame can be obtained, the motion trajectory sequence is composed of position coordinates of the feature points, and the points on the motion trajectory sequence are also feature points.

[0052] It should be noted that the target detection algorithm (such as YOLO) and the corner detection algorithm are both known technologies, and the specific process is not described here.

[0053] The collection and acquisition of personal information data in the embodiment of the present application are all authorized by the relevant users, and the process does not violate the relevant laws and regulations and does not violate the public order and good customs.

[0054] Step S2: In the motion image frame where the joint is missing, all feature points on the joint edge to which the missing joint belongs are taken as reference points; for each missing joint, the position change difference features of the reference points in the motion trajectory sequence are analyzed to determine the effective factors of each reference point; the position fitting smoothness between the reference points is analyzed to determine the smooth factors of each reference point on the joint edge; and the prediction position of each missing joint is obtained based on the effective factors, the smooth factors of the reference points corresponding to each missing joint, and the position distribution of the reference points.

[0055] Since the movement of the joint edge in the human body fitness action has certain rules, the position of the unknown joint node can be constrained. Therefore, in the embodiment of the present application, in the motion image frame in which the joint node is missing, all feature points on the joint edge to which the missing joint node belongs are taken as reference points of the missing joint node, and the spatial relationship between the joint edge and the joint node is converted into mathematical constraints in the subsequent process, so that the accuracy in the subsequent position prediction process can be improved, and it is ensured that the predicted position can be more consistent with the human body structure.

[0056] The human body limb movement is usually continuous, for example, the movement of the forearm can be regarded as the movement of a rigid link, that is, the relative movement vector between the elbow and the wrist is smoothly changed in a movement process. Assuming that the movement of the joint edge is linear, the movement trajectory of the reference point on the joint edge can reflect the movement vector constituted by the two continuous motion image frames, which has direction and module length and is linearly changed. Among all the reference points, there may be some points with coordinate deviation due to the violent shaking of the human body. Therefore, the reference value of each reference point needs to be analyzed, which can be specifically represented by an effective factor and a smoothing factor of the reference point.

[0057] When the reference point corresponding to the missing joint node has better continuity in the tracking process of the movement trajectory sequence, the effective degree will be higher, and the reference value will be larger. Therefore, for each missing joint node, the position change difference feature of the reference point in the movement trajectory sequence is analyzed to determine the effective factor of each reference point, which is used to represent the continuity in the tracking process.

[0058] Preferably, in an embodiment of the present application, the method for obtaining the effective factor comprises:

[0059] The motion image frame in which the joint node is missing is taken as a target frame, in each target frame, the position change in the motion trajectory sequence of each reference point of the missing joint node is analyzed, the motion vector of the reference point under each motion image is obtained, and two features of each reference point are determined, the two features are a module length and an angle value respectively: in each target frame, in the motion trajectory sequence of each reference point of the missing joint node (the motion trajectory sequence is composed of coordinate positions of pixel points in continuous motion image frames, and reflects the motion trajectory of the reference point), each coordinate position is taken as an end point, and the adjacent previous coordinate position is taken as a start point, so that the motion vector of the reference point under each motion image frame (not including the first motion image frame) is obtained, the module length of the motion vector is taken as the first feature of the reference point, and the angle value between the motion vector of the reference point under each motion image and a preset direction is taken as the second feature of the reference point, in the embodiment of the application, the preset direction is a horizontal right direction, and the angle value between the motion vector and the preset direction is an angle rotated clockwise from the preset direction to the motion direction. By calculating the module length and the angle value corresponding to the motion vector, the dynamic characteristics can be better captured.

[0060] The standard deviation can measure the volatility of a group of data, the larger the standard deviation, the more unstable the data, therefore, for any one feature of each reference point, the standard deviation of all the values of each reference point under the feature is taken as the first invalid parameter of each reference point, based on the foregoing analysis, the larger the first invalid parameter, the more unstable the position change feature of the reference point in the motion trajectory sequence, and the lower the reference value of the reference point; then the deviation feature between the change of the motion vector and the whole is continuously measured, the absolute value of the difference between the value of each reference point under the feature in the target frame and the average value of the value of each reference point under the feature in the motion trajectory sequence is taken as the second invalid parameter of each reference point, the larger the second invalid parameter, the greater the deviation degree between the value of the reference point under the feature in the target frame and the whole of the motion trajectory sequence, and the lower the stability, and the lower the reference value. Therefore, the first invalid parameter and the second invalid parameter of each reference point are multiplied, and the obtained product is negatively correlated and normalized to correct the logical relationship, and the effective factor corresponding to each reference point under the feature is obtained, at this time, the larger the effective factor, the more stable the change feature of the reference point in the whole motion trajectory sequence, and the higher the reference value of the reference point, and the higher the accuracy of subsequent position prediction according to the reference point. The negative correlation mapping and normalization processing can adopt the formula wherein, represents an exponential function with a natural constant e as a base, and x represents an independent variable.

[0061] Since the motion pattern of the reference point existing on the same joint edge should have a more regular change trend, for each missing joint node, the smoothness of the position fitting between all the reference points on the joint edge where the missing joint node is located can be analyzed, so as to quantify the smoothness factor of each reference point on the joint edge as another index reflecting the reference value of the reference point.

[0062] Preferably, the method for obtaining the smoothness factor in an embodiment of the present application comprises:

[0063] For any feature of the reference point, the values of all the reference points of each missing joint node in the feature are curve-fitted according to the arrangement order on the joint edge to obtain a first fitting curve, the first fitting curve maps the discrete points to the continuous function space, and the motion trend of the reference point can be extracted through the smooth processing.

[0064] Then, the mean square error between the first fitting curve and the values of all the reference points in the feature is calculated, the greater the mean square error, the greater the difference between the actual values of the reference points and the first fitting curve, the more unsmooth the motion trajectory of the reference points as a whole, the more likely there is abnormal motion, and the lower the reference value; on the contrary, the smaller the mean square error, the closer the actual values of the reference points to the first fitting curve, the more smooth the motion trajectory of the reference points as a whole, so the mean square error is negatively correlated and normalized to correct the logical relationship, obtain the smoothness factor, and the smoothness factor is taken as the smoothness factor of each reference point corresponding to each missing joint node, the greater the smoothness factor, the higher the reference value of the reference point corresponding to the missing joint node, and the higher the accuracy of the subsequent position prediction based on the reference point. The negative correlation and normalization processing can be performed by the formula wherein, represents the exponential function with the natural constant e as the base, and x represents the independent variable.

[0065] The effective factor and the smoothness factor of the reference point calculated in the foregoing steps can both reflect the credibility or reference value of predicting the position of the missing joint node based on the reference point, so based on the foregoing two indexes and in combination with the position distribution of the reference point, the predicted position of each missing joint node can be obtained.

[0066] Preferably, the method for obtaining the predicted position in an embodiment of the present application comprises:

[0067] In each target frame, an optional non-missing joint node on the joint edge to which the missing joint node belongs (that is, a joint node that does not have a missing condition in the target frame) is taken as a reference point, the normalized value of the Euclidean distance between each reference point and the reference point is taken as a distance parameter, thereby the position information of the reference point can be converted into relative distance, so that a specific numerical value that can be quantified can be provided, which is convenient for subsequent calculation. The normalization is a technique known to those skilled in the art, and the selection of the normalization function can be linear normalization or standard normalization, and the specific normalization method is not limited here.

[0068] The effective factor and the smoothing factor of the reference point under each feature are combined to obtain the confidence of the reference point under each feature. Since the effective factor and the smoothing factor are positively correlated with the reference value of the reference point, the product of the effective factor and the smoothing factor of each reference point under each feature of the reference point is normalized to obtain the confidence of each reference point under each feature. The greater the confidence, the higher the contribution of the reference point to the accuracy of predicting the position of the missing joint node. The normalization is a technique known to those skilled in the art, and the selection of the normalization function can be linear normalization or standard normalization, and the specific normalization method is not limited here.

[0069] In each feature of the reference point, the distance parameter corresponding to the reference point is taken as the horizontal axis, the value of the reference point under each feature is taken as the vertical axis, and the confidence of the reference point under each feature is taken as the weight, thereby the curve fitting is performed to obtain the second fitting curve of each feature. By taking the confidence as the weight, the second fitting curve can be more dependent on the reference point with high reference value, and the influence of the abnormal reference point can be reduced.

[0070] Then in other motion image frames before the target frame time sequence, the motion image frames in which the missing joint node in the target frame and the reference point do not have missing are taken as comparison frames, in the comparison frame closest to the target frame time sequence, the joint node of the same type as the missing joint node is taken as an analysis point, and the normalized value of the Euclidean distance between the analysis point and the joint node of the same type as the reference point is mapped in the second fitting curve under each feature, thereby obtaining the corresponding value of the missing joint node in the target frame under each feature.

[0071] Finally, the predicted position of the missing joint node in the target frame can be calculated according to the length of the motion vector, the angle value and the position coordinates of the analysis point corresponding to the missing joint node in the target frame.

[0072] The process is illustrated as follows: for example, there are 5 motion image frames, the missing joint in the 4th motion image frame is the left knee joint, the left ankle joint is selected as the reference point (the left ankle joint and the left knee joint belong to the same joint edge of the lower leg), the 4th motion image frame is the target frame, if the left knee joint and the left ankle joint in the 2nd and 3rd motion image frames are not missing, the 3rd motion image frame closest in time is selected as the comparison frame, and the left knee joint in the 3rd motion image frame is the analysis point, the normalized value of the Euclidean distance between the left knee joint and the left ankle joint in the 2nd motion image frame is mapped under the second fitting curve of each feature, so that the corresponding values of the left knee joint in the 4th motion image frame under each feature, that is, the length of the motion vector corresponding to the missing joint and the angle value, can be obtained, and in the case that the position coordinates of the analysis point, that is, the left knee joint in the 3rd motion image frame, are known, the specific position coordinates of the missing joint in the 4th motion image frame can be calculated according to the length of the motion vector and the angle value, and the specific position coordinates are taken as the predicted position of the missing joint.

[0073] It should be noted that the curve fitting in the embodiments of the present application can be obtained according to the least square method, and the specific process is not repeated here.

[0074] Step S3: In the first motion image frame, the matching motion trajectory sequence of each joint is determined based on the positional relationship between the joints and the feature points; and based on the predicted position of the missing joint, the consistency degree of the change of the matching motion trajectory sequence between the missing joint and other joints is analyzed, so as to determine the final position of the missing joint.

[0075] By the positional relationship between the joints and the feature points in the first frame (when the exercise just starts), the relative spatial layout of the joints and the feature points is determined, the kinematics constraint of the human body can be formed, and the final result is ensured to be more consistent with the human body structure, so in the first motion image frame, the matching motion trajectory sequence of each joint is determined based on the positional relationship between the joints and the feature points.

[0076] Preferably, in an embodiment of the present application, the method for obtaining the matching motion trajectory sequence comprises:

[0077] In the first motion image frame, the Euclidean distance between each joint and each feature point is calculated as a distance factor between each joint and each feature point.

[0078] The smaller the distance factor is, the closer the position between the joint and the feature point is, and then the stronger the motion correlation is, so for any joint, the motion trajectory sequence of the feature point with the smallest distance factor and the matching motion trajectory sequence of the joint are selected as the matching motion trajectory sequence of the joint.

[0079] It should be noted that in this embodiment of the present application, the matching motion trajectory of each joint point is determined in the first motion image frame, that is, the matching motion trajectory sequence of all left ankle joints (including the case where the left ankle joint is missing) in the subsequent motion image frames is determined.

[0080] In step S2, the predicted position of the missing joint point has been obtained, and based on the predicted position, the consistency degree of the change of the matching motion trajectory sequence between the missing joint point and other joint points can be analyzed, and the position of the missing joint point is further constrained by using the coordination constraint of human motion, so as to obtain a more accurate final position.

[0081] Preferably, the method for obtaining the final position in an embodiment of the present application comprises:

[0082] Human fitness actions (such as push-ups, sit-ups, etc.) will all cause the joint motion to exhibit a repetitive and periodic characteristic, so by verifying the periodic consistency degree of the matching motion trajectory, abnormal data that does not conform to the motion law can be filtered out, and the dynamic continuity of the final position of the missing joint point and the scene rationality are ensured.

[0083] Therefore, the predicted position of the missing joint point is used to replace the coordinates of the feature points belonging to the target frame in the corresponding matching motion trajectory, so as to obtain an updated motion trajectory, and the horizontal and vertical coordinates of all coordinates in the updated motion trajectory are sequentially subjected to Fourier transformation, and the periods corresponding to the main peak frequencies of the horizontal and vertical coordinates are obtained respectively (the period is the inverse of the frequency) ; the horizontal and vertical coordinates of all feature points in the matching motion trajectory of the reference point corresponding to the missing joint point are sequentially subjected to Fourier transformation, and the periods corresponding to the main peak frequencies of the horizontal and vertical coordinates are obtained respectively.

[0084] Then, the absolute value of the difference between the periods corresponding to the main peak frequencies of the horizontal coordinates of the missing joint point and the corresponding reference point is calculated as a first difference factor, and the absolute value of the difference between the periods corresponding to the main peak frequencies of the vertical coordinates of the missing joint point and the corresponding reference point is calculated as a second difference factor; the first difference factor evaluates the motion synchronization in the left-right direction, and the second difference factor evaluates the motion synchronization in the up-down direction, so when the first difference factor is larger and the second difference factor is larger, it means that the motion is less synchronized, that is, the accuracy of the predicted position is lower.

[0085] Therefore, if the first difference factor and the second difference factor corresponding to the missing joint node are both less than the preset difference threshold, it is indicated that the predicted position is possibly close to the true value, and further correction is not needed, and the predicted position of the missing joint node is taken as the final position. Otherwise, the median value (the median value of the horizontal and vertical coordinates is calculated respectively) of the coordinates of the feature points belonging to the target frame and the coordinates of the predicted position is taken as the final position of the missing joint node in the matching motion trajectory sequence of the missing joint node.

[0086] It should be noted that the Fourier transform is a known technology, and the specific process is not described here; in this embodiment of the present application, the preset difference threshold is 0.5, and the specific value can be adjusted according to the implementation scene, which is not limited here.

[0087] Therefore, through the above process, the missing joint node in the continuous motion image frame can be supplemented, and more continuous motion change data is obtained, and the action recognition and action quality judgment based thereon will also be more accurate.

[0088] In summary, the continuous motion image frames of the to-be-tested person are collected, the joint nodes and joint edges of the human body skeleton in each motion image frame are obtained, and the motion trajectory sequence of the feature points in the motion image frame is tracked, which is used for compensating and analyzing the motion change characteristics of the joint nodes when the joint nodes are missing in the subsequent process through the association of the feature points. First, in the motion image frame in which the joint nodes are missing, all the feature points on the joint edge to which the missing joint node belongs are used as reference points, and the effective factor and the smoothing factor are determined by combining the position change difference characteristics of the reference points in the motion trajectory sequence. Then, the effective factor, the smoothing factor and the spatial distribution law of the reference points are fused, so that the predicted position of the missing joint node is more consistent with the linear constraint of the human joint edge, and the limb distortion error that may be caused by the traditional interpolation method is eliminated. Finally, in the first motion image frame, the joint nodes and the feature points are associated based on the positional relationship between the joint nodes and the feature points, so as to determine the matching motion trajectory sequence of each joint node, and then based on the predicted position of the missing joint node, the consistency degree of the matching motion trajectory sequence of the missing joint node and the adjacent joint is analyzed, so as to ensure that the final position of the missing joint node is more consistent with the human kinematics law, improve the physical rationality of the motion change characteristics, and ensure the accuracy of the motion data acquisition.

[0089] It should be noted that the above-mentioned embodiment of the present application is only for description, and does not represent the advantages and disadvantages of the embodiment. The process depicted in the drawing does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or may be advantageous.

[0090] The various embodiments described in this specification are presented by way of example, and each embodiment is not inherently more important than any other embodiment.

Claims

1. A fitness motion data collection and analysis method based on a video deep learning algorithm, characterized in that, The method comprises: In the continuous motion image frames of the to-be-tested person, the human skeleton detection result in each motion image frame and the motion trajectory sequence of each feature point are acquired, wherein the human skeleton detection result comprises joint edges and joint nodes; In the motion image frame in which the joint node is missing, all feature points on the joint edge to which the missing joint node belongs are taken as reference points; for each missing joint node, the position change difference features of the reference points in the motion trajectory sequence are analyzed, the effective factors of each reference point are determined; the position fitting smoothness between the reference points is analyzed, the smoothness factors of each reference point on the joint edge are determined; the prediction positions of each missing joint node are acquired based on the effective factors, the smoothness factors of the reference points corresponding to each missing joint node and the position distribution of the reference points; In the first motion image frame, the matching motion trajectory sequence of each joint node is determined based on the position relationship between the joint nodes and the feature points; based on the prediction positions of the missing joint nodes, the change consistency degrees of the matching motion trajectory sequences between the missing joint nodes and other joint nodes are analyzed, so that the final positions of the missing joint nodes are determined; The acquisition method of the effective factors comprises: The motion image frame in which the joint node is missing is taken as a target frame; In each target frame, the position change of each reference point of the missing joint node in the motion trajectory sequence is analyzed, the motion vectors of the reference points in each motion image frame are acquired, so that two features of each reference point are determined, the two features are respectively a modulus and an angle value; For any one feature of each reference point, the standard deviation of all values of each reference point in the feature is taken as a first invalid parameter of each reference point, the absolute value of the difference between the value of each reference point in the feature in the target frame and the average value of the values of each reference point in the feature in the motion trajectory sequence is taken as a second invalid parameter of each reference point, and the value obtained by performing negative correlation mapping and normalization on the product of the first invalid parameter and the second invalid parameter of each reference point is taken as the effective factor corresponding to the feature of each reference point. The acquisition method of the smoothness factors comprises: For any one feature of the reference points, the values of all reference points of each missing joint node in the feature are curve-fitted according to the arrangement order on the joint edge, so that a first fitting curve is obtained; The mean square error between the first fitting curve and the values of all reference points in the feature is calculated, and negative correlation mapping and normalization processing are performed, so that the smoothness factor of each reference point corresponding to each missing joint node is obtained.

2. The method of claim 1, wherein the method is based on a video deep learning algorithm. The acquisition method of the motion trajectory sequence of each feature point comprises: Based on a target detection algorithm, the human body region is framed in each motion image frame, and based on a corner point detection algorithm, the corner points are acquired as feature points in the human body region; Based on an optical flow method, the positions of each feature point in the first motion image frame are tracked in all motion image frames, so that the motion trajectory sequence of each feature point is obtained, and the points on the motion trajectory sequence are all taken as feature points. 3.The method of claim 1, wherein, The position change in the motion trajectory sequence of each reference point missing the joint node is analyzed in each target frame to obtain a motion vector of the reference point under each motion image frame, thereby determining two features of each reference point, including: In each target frame, in the motion trajectory sequence of each reference point missing the joint node, each coordinate position is taken as an end point, and the adjacent previous coordinate position is taken as a start point, thereby obtaining a motion vector of the reference point under each motion image, taking the module length of the motion vector as a first feature of the reference point, and taking an angle value between the motion vector of the reference point under each motion image and a preset direction as a second feature of the reference point.

4. The method of claim 1, wherein the method further comprises: The method for obtaining the predicted position comprises: In each target frame, an optional non-missing joint node on the joint edge to which the missing joint node belongs is taken as a reference point, and a normalized value of a Euclidean distance between each reference point and the reference point is obtained as a distance parameter; The effective factor and the smoothing factor of the reference point under each feature are combined, thereby obtaining a confidence of the reference point under each feature; Under each feature of the reference point, the distance parameter corresponding to the reference point is taken as a horizontal axis, the value of the reference point under each feature is taken as a vertical axis, and the confidence of the reference point under each feature is taken as a weight, thereby performing curve fitting to obtain a second fitting curve of each feature; In other motion image frames before the time sequence of the target frame, a motion image frame in which the missing joint node and the reference point in the target frame are both not missing is taken as a comparison frame, in the comparison frame closest in time sequence to the target frame, a joint node of the same type as the missing joint node is taken as an analysis point, and a normalized value of a Euclidean distance between the analysis point and a joint node of the same type as the reference point is mapped in the second fitting curve under each feature, thereby obtaining a corresponding value of the missing joint node in the target frame under each feature; According to the module length, the angle value of the motion vector of the missing joint node in the target frame, and the position coordinates of the analysis point, the predicted position of the missing joint node in the target frame is calculated.

5. The method of claim 4, wherein the method further comprises: The method for obtaining the confidence comprises: In each feature of the reference point, a product of the effective factor and the smoothing factor of each reference point is normalized, and the normalized value is taken as the confidence of each reference point under each feature.

6. The method of claim 4, wherein the method further comprises: The method for obtaining the final position comprises: The predicted position of the missing joint node is used to replace the coordinates of the feature points belonging to the target frame in the corresponding matching motion trajectory, thereby obtaining an updated motion trajectory, and the horizontal and vertical coordinates of all coordinates in the updated motion trajectory are sequentially subjected to Fourier transformation, thereby obtaining a period corresponding to the main peak frequency of the horizontal and vertical coordinates, respectively; The horizontal and vertical coordinates of all feature points in the matching motion trajectory of the reference point corresponding to the missing joint node are sequentially subjected to Fourier transformation, thereby obtaining a period corresponding to the main peak frequency of the horizontal and vertical coordinates, respectively; The absolute value of the difference between the periods corresponding to the main peak frequencies of the horizontal coordinates of the missing joint node and the reference point corresponding thereto is taken as a first difference factor, and the absolute value of the difference between the periods corresponding to the main peak frequencies of the vertical coordinates of the missing joint node and the reference point corresponding thereto is taken as a second difference factor. If the first difference factor and the second difference factor corresponding to the missing joint node are both less than the preset difference threshold, the predicted position of the missing joint node is taken as the final position, otherwise, the median value of the coordinates of the feature points belonging to the target frame and the predicted position is taken as the final position of the missing joint node in the matching motion trajectory sequence of the missing joint node. 7.The method of claim 1, wherein, The method for obtaining the matching motion trajectory sequence comprises: In the first motion image frame, the Euclidean distance between each joint node and each feature point is calculated as the distance factor between each joint node and each feature point. For any joint node, the motion trajectory sequence of the feature point with the minimum distance factor and the matching motion trajectory sequence of the joint node are taken as the matching motion trajectory sequence of the joint node in all distance factors corresponding to the joint node. 8.The method of claim 6, wherein the method further comprises: determining a plurality of body parts of the user based on the plurality of images; and determining a plurality of body part positions of the user based on the plurality of images. The difference threshold is 0.5.

Images