A home fitness intelligent auxiliary system
By combining data collection, extraction, construction, and evaluation units, the problem of non-standard movements in home fitness is solved, enabling accurate identification of exercise types and effect evaluation, thus improving the accuracy and safety of fitness results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-14
- Publication Date
- 2026-03-24
AI Technical Summary
In home fitness, the lack of professional guidance can lead to incorrect movements, difficulty in identifying exercise types and assessing exercise effects, and may cause physical injury.
The system uses an acquisition unit to acquire real-time three-dimensional position information of the moving object, a real-time motion feature extraction unit to extract motion feature sequences, a standard motion feature construction unit to construct standard motion feature sequences, an evaluation unit to identify motion type and effect, and a display unit to show the results.
It enables accurate detection and recognition of home fitness movements, improves the accuracy of exercise effect evaluation, and avoids physical injury caused by incorrect movements.
Smart Images
Figure CN116343325B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of motion detection, and specifically provides a home fitness intelligent auxiliary system. BACKGROUND
[0002] With the continuous improvement of living standards and health awareness, various fitness exercises have become an essential activity in the lives of more and more people. At the same time, under the background of epidemic prevention and control, fitness exercises in the home and indoor environment are necessary and superior. Regularly performing various home fitness exercises not only helps to improve physical fitness and enhance resistance to various diseases, but also can relax and enjoy the mood, effectively alleviate various psychological problems, so home fitness has become a popular exercise method.
[0003] However, due to the variety of exercises that can be performed during home fitness, without professional training or personal guidance, the actions may not be standard or in place, which not only fails to achieve the fitness effect, but also may cause damage to the body due to incorrect movements.
[0004] Therefore, it is necessary to provide a system that can accurately and effectively detect and identify the actions of home fitness exercises and evaluate the exercise effect based on the identification results. SUMMARY
[0005] To solve the problems existing in the prior art, the application provides a home fitness intelligent auxiliary system, which comprises a collection unit, a real-time action feature extraction unit, a standard action feature construction unit, an evaluation unit, a database and a display unit.
[0006] The collection unit is used to collect real-time three-dimensional position information of at least one motion object.
[0007] The real-time action feature extraction unit extracts a real-time action feature sequence of at least one motion object based on the real-time three-dimensional position information.
[0008] The standard action feature construction unit is used to construct a standard action feature sequence of at least one type of motion and save it in the database.
[0009] The evaluation unit identifies the motion type and motion effect of at least one motion object based on the real-time action feature sequence and the standard action feature sequence.
[0010] The display unit is used to display the real-time three-dimensional position information, the motion type and the motion effect.
[0011] Further, the real-time action feature extraction unit comprises a coordinate extraction module, a number of people recognition module, a multi-person segmentation module, a processing module; the real-time action feature extraction unit extracts the real-time action feature sequence of at least one moving object based on the following steps:
[0012] S100: The coordinate extraction module determines the real-time three-dimensional coordinates and real-time visibility of the plurality of joint nodes based on the real-time three-dimensional position information;
[0013] S200: The number of people recognition module recognizes the number of moving objects based on the real-time three-dimensional coordinates of the plurality of joint nodes;
[0014] S300: If the number of moving objects is greater than 1, execute step S400, otherwise execute step S500;
[0015] S400: The multi-person segmentation module segments the plurality of joint nodes based on the number of moving objects, and executes step S500 for each moving object after segmentation;
[0016] S500: The processing module processes to generate the real-time action feature sequence of each moving object, the real-time action feature sequence comprising a real-time action feature set of the moving object at each time point, wherein the real-time action feature set at each time point comprises the real-time three-dimensional coordinates and real-time visibility of each joint node of the moving object at the time point.
[0017] Preferably, the plurality of joint nodes comprises a plurality of facial joint nodes, a plurality of torso joint nodes, and a plurality of limb joint nodes.
[0018] Preferably, the number of people recognition module determines the number of moving objects using at least one of the following criteria: the number of faces determined based on the number of facial joint nodes, the total number of joint nodes, and the three-dimensional distance between any two joint nodes.
[0019] Preferably, the real-time action feature extraction unit further comprises an angle extraction module, which determines the real-time deflection angle of each joint node based on the real-time three-dimensional coordinates of the plurality of joint nodes, and the real-time action feature set at each time point further comprises the real-time deflection angle of each joint node of the moving object at the time point.
[0020] Preferably, the real-time action feature extraction unit further comprises a posture correction module, which corrects the real-time action feature sequence based on the real-time three-dimensional coordinates of the plurality of torso joint nodes.
[0021] Further, the standard action feature construction unit constructs the standard action feature sequence of each type of movement based on the following steps:
[0022] A100: obtaining a standard motion video of the type of motion;
[0023] A200: calibrating a plurality of reference joints from the standard motion video of the type of motion;
[0024] A300: selecting a real-time motion feature sequence with the same shooting angle as the standard motion video from each real-time motion feature sequence as a candidate motion feature sequence;
[0025] A400: adjusting the positions of corresponding joints in the candidate motion feature sequence based on the positions of the plurality of reference joints;
[0026] A500: adjusting the positions of non-corresponding joints in the candidate motion feature sequence based on the positions of corresponding joints in the candidate motion feature sequence;
[0027] A600: generating a standard motion feature sequence of the type of motion based on the positions of corresponding joints and the positions of non-corresponding joints, the standard motion feature sequence comprising a plurality of standard motion feature sets at a plurality of time points, wherein each standard motion feature set at a time point comprises standard three-dimensional coordinates and standard visibility of each joint at the time point.
[0028] Preferably, the standard visibility is determined based on the plurality of reference joints calibrated from the standard motion video.
[0029] Preferably, the evaluation unit uses a trained deep learning network to identify the type of motion of the at least one moving object.
[0030] Further, the evaluation unit identifies the type of motion and the motion effect of the at least one moving object based on the following steps:
[0031] B100: extracting a real-time motion feature sequence of the at least one moving object;
[0032] B200: identifying the type of motion of the moving object based on the trained deep learning network;
[0033] B300: extracting a standard motion feature sequence corresponding to the identified type of motion;
[0034] B400: correcting the standard visibility of each joint in the standard motion feature sequence corresponding to the identified type of motion based on the posture of the real-time motion feature sequence of the moving object;
[0035] B500: count the number of different visible points between the real-time visible points of the motion object and the standard visible points of the corrected standard motion feature sequence, if the count is greater than a first threshold, the motion effect is evaluated as poor and returns to step B100, otherwise, step B600 is executed;
[0036] B600: calculate the matching degree between the real-time motion feature sequence of the motion object and the corrected standard motion feature sequence, if the matching degree is less than a second threshold, the motion effect is evaluated as general and returns to step B100, otherwise, step B700 is executed;
[0037] B700: the motion effect is evaluated as good and returns to step B100.
[0038] The home fitness intelligent auxiliary system provided by the embodiments of the present application extracts the real-time three-dimensional positions and visible points of multiple joints from the real-time motion information of the motion object, can identify the number of motion objects in real time, and identify and evaluate the motion type and motion effect of each motion object; at the same time, the real-time motion feature sequence of the motion object is corrected using the 2D image material of the standard motion, to obtain the standard motion feature sequence, various types of standard motion feature sequences are conveniently and accurately generated on the basis of not increasing the system cost, and the accuracy of identifying the motion type and evaluating the motion effect is effectively improved. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 FIG. 1 is a schematic diagram of the architecture of the home fitness intelligent auxiliary system according to the embodiments of the present application;
[0040] Figure 2 FIG. 2 is a real-time 2D image of a specific motion object performing motion according to the embodiments of the present application;
[0041] Figure 3 FIG. 3 is a schematic diagram of the architecture of the real-time motion feature extraction unit according to the embodiments of the present application;
[0042] Figure 4 FIG. 4 is a flowchart of the coordinate extraction module extracting the real-time motion feature sequence of the motion object according to the embodiments of the present application;
[0043] Figure 5 FIG. 5 is a schematic diagram of the real-time positions of multiple joints of the motion object extracted by the coordinate extraction module according to the embodiments of the present application;
[0044] Figure 6 FIG. 6 is a schematic diagram of the standard three-dimensional coordinates of each joint constructed by the standard motion feature construction module according to the embodiments of the present application;
[0045] Figure 7a ,Figure 7b are respectively schematic diagrams of the same type of movement with different postures of the moving object according to an embodiment of the present application. DETAILED DESCRIPTION
[0046] Hereinafter, the present application will be further described based on preferred embodiments and with reference to the accompanying drawings.
[0047] The words in the present specification are used for explaining the embodiments of the present application, but are not intended to limit the present application. It should also be noted that, unless explicitly specified and limited, if the terms "arranged", "connected", "linked" appear, they should be interpreted broadly, for example, can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be directly connected, or indirectly connected through an intermediate medium, can be the internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be specifically understood.
[0048] The present application provides a home fitness intelligent auxiliary system by embodiments, Figure 1 The architecture schematic diagram of the home fitness intelligent auxiliary system in some embodiments is shown. As shown in the figure, Figure 1 The home fitness intelligent auxiliary system includes a collection unit, a real-time action feature extraction unit, a standard action feature construction unit, an evaluation unit, a database and a display unit. The specific embodiments of each unit will be described in detail below in conjunction with the drawings.
[0049] In the embodiments of the present application, the collection unit is used to collect real-time three-dimensional position information of at least one moving object. At present, there are various devices that can collect real-time three-dimensional position information of moving objects, for example, through a camera with a depth sensor, real-time 2D images of moving objects and depth information of different parts of their bodies relative to the camera can be obtained along the shooting angle; for example, through a binocular camera that simulates the interpupillary distance of the human eye, the moving object can be photographed at different angles, so as to obtain image pairs with parallax, and by calculating the parallax at different positions in the image pair, the depth information of different parts of the moving object's body relative to the binocular camera can also be obtained. The above various ways of collecting real-time three-dimensional position information of moving objects are well known to those skilled in the art. Figure 2 The real-time 2D images of the moving object obtained by the collection unit in some specific embodiments of the present application are schematically shown.
[0050] In the embodiments of the present application, the real-time action feature extraction unit extracts a real-time action feature sequence of at least one moving object based on the above-mentioned real-time three-dimensional position information. Specifically, as shown in Figure 3As shown, the real-time action feature extraction unit includes a coordinate extraction module, a number of people recognition module, a multi-person segmentation module, and a processing module. Those skilled in the art should know that the real-time action feature extraction unit can have multiple implementation manners, for example, in some specific embodiments, each module of the above real-time action feature extraction unit can be a program module stored in a hard disk, a flash memory or the like, and is called and executed by a central processing unit; in another specific embodiment, each program module can also be implanted in a Jetson Nano or the like integrated development board specially for AI application.
[0051] Figure 4 A flowchart showing that in some specific embodiments, the real-time action feature extraction unit extracts a real-time action feature sequence of at least one moving object is shown. As Figure 4 As shown, the real-time action feature extraction unit extracts a real-time action feature sequence of at least one moving object based on the following steps:
[0052] S100: The coordinate extraction module determines real-time three-dimensional coordinates and real-time visibility of a plurality of joint nodes based on the real-time three-dimensional position information;
[0053] S200: The number of people recognition module recognizes the number of moving objects based on the real-time three-dimensional coordinates of the plurality of joint nodes;
[0054] S300: If the number of moving objects is greater than 1, step S400 is executed, otherwise step S500 is executed;
[0055] S400: The multi-person segmentation module segments the plurality of joint nodes based on the number of moving objects, and executes step S500 for each moving object after segmentation;
[0056] S500: The processing module processes to generate a real-time action feature sequence of each moving object, the real-time action feature sequence including a real-time action feature set of the moving object at a plurality of time points, wherein the real-time action feature set at each time point includes real-time three-dimensional coordinates and real-time visibility of each joint node of the moving object at the time point.
[0057] Specifically, in step S100, the coordinate extraction module extracts real-time three-dimensional coordinates of a plurality of joint nodes of a moving object from three-dimensional position information acquired by the acquisition unit to represent posture information of the moving object when performing various actions. Since the muscles, skin tissue and the like of the human body are supported by the skeleton, and each bone is connected by the joint, the spatial positions between the plurality of joint nodes of the human body can accurately describe the posture features of the human body.
[0058] The coordinate extraction module can extract a plurality of joints of the moving object and track the real-time three-dimensional coordinates of each joint using a machine learning framework known to those skilled in the art such as MediaPipe. Figure 5 A diagram showing the plurality of joints extracted by the coordinate extraction module to represent the posture of the moving object in one specific embodiment is shown. In this embodiment, the coordinate extraction module extracts a total of 33 joints, including 11 facial joints, 4 trunk joints, and 18 limb joints. Table 1 below lists the specific positions of the 33 joints in the diagram, respectively.
[0059] Table 1
[0060]
[0061]
[0062] The positional relationship between the above-mentioned 33 joints will change differently when the moving object performs various types of movements. For example, the positional relationship between the various facial joints remains basically unchanged when performing various movements; the relative positional relationship of the various trunk joints will exhibit different trends when performing twisting and bending movements; and the relative positional relationship of the various limb joints is more flexible and variable. By utilizing the changes in the relative positions between different joints, the body posture of the moving object when performing various movements can be represented. In addition, it should be understood that those skilled in the art can flexibly add or delete joints to meet the needs of movement recognition and evaluation. Figure 5 A diagram showing the real-time positions of the plurality of joints of the moving object extracted by the coordinate extraction module in one specific embodiment is shown.
[0063] In addition, during the extraction and tracking of the above-mentioned various joints using a machine learning framework such as MediaPipe, there may be cases where some joints are blocked due to specific movements of the moving object. Therefore, in the embodiments of the present application, the coordinate extraction module first extracts and tracks the real-time three-dimensional coordinates of a plurality of visible joints, and estimates the real-time three-dimensional coordinates of a plurality of invisible joints based on the real-time three-dimensional coordinates of the visible joints to obtain the real-time three-dimensional coordinates of all joints. At the same time, an additional flag is added to represent the real-time visibility of each joint (for example: when a joint is visible in the real-time 2D image obtained by the acquisition unit, its real-time visibility is set to 1, otherwise it is set to 0).
[0064] Specifically, step S200 is configured to identify the number of motion objects performing exercise simultaneously. Since there can be multiple motion objects performing exercise simultaneously in a home exercise process, the number of joint points extracted and tracked by the coordinate extraction module and the spatial distribution are obviously different from the case of a single motion object performing exercise. Therefore, the number of motion objects needs to be identified first, and the real-time motion feature sequence is identified after segmentation of different motion objects, so as to ensure the accuracy of identification.
[0065] In some specific embodiments, the number of motion objects is determined by the number of faces based on the number of face joint points, the total number of joint points, and the three-dimensional distance between any two joint points. By extracting and tracking the number of face joint points, the number of motion objects can be accurately identified when each motion object faces the acquisition unit. By using the total number of visible joint points and the three-dimensional distance between any two joint points as criteria, the number of motion objects identified by face joint points can be corrected when multiple motion objects overlap or are blocked in the depth direction, so as to accurately identify the number of motion objects. For example, in the case that the face of a motion object is blocked, the total number of visible joint points can be counted to effectively determine whether the number of motion objects is more than one. For another example, when the three-dimensional distance between any two joint points is obviously greater than the normal height of a human body (e.g., the three-dimensional distance between two joint points is more than 2 meters), it can be determined that there are at least two motion objects.
[0066] Specifically, if the number of identified motion objects is more than one, the multiple joint points extracted are segmented by the multi-person segmentation module according to the number of identified motion objects in step S400. The segmentation can use a machine learning framework known to those skilled in the art. For example, in one specific embodiment, the multi-person segmentation module uses a trained YOLO target recognition framework to segment multiple motion objects. The specific steps include:
[0067] Step one, dividing the real-time 2D image obtained by the acquisition unit into cells with a side length of S;
[0068] Step two, predicting the bounding box Bbox and the confidence based on the number of motion objects by full convolutional neural network CNN calculation, and calculating the posterior probability of the Bbox in which a human body is predicted to exist;
[0069] Step three, selecting the bounding box Bbox in descending order of the posterior probability to segment the real-time 2D image and the corresponding depth image, wherein each bounding box includes the joint points of a motion object.
[0070] The YOLO target recognition framework needs to be trained to be applied to segmentation of multiple moving objects. When the YOLO target recognition framework is trained, excessive joint nodes moving out of the line of sight can cause deformation of the skeleton posture and affect the training effect. Therefore, when the YOLO target recognition framework is trained, the confidence of each joint node in the training set needs to be considered additionally. Specifically, when a joint node is in the line of sight (regardless of whether it is visible), its confidence is set to 1; when a joint node is out of the line of sight, its confidence is 0. The average of the confidence of all joint nodes is calculated, and data with an average confidence less than 0.5 is removed from the training set.
[0071] Finally, after completing the number recognition of the moving object and the segmentation of the joint nodes of the multiple moving objects, the real-time action feature sequence of each moving object can be extracted by the processing module in step S500. Specifically, the real-time action feature sequence includes a real-time action feature set of the moving object at multiple time points, and each real-time action feature set at a time point includes real-time three-dimensional coordinates and real-time visibility of each joint node of the moving object at the time point. For example, for a specific moving object Object1, each time point t1, t2, …, t i ,…,t n in the time sequence formed by the multiple time points t i has a real-time action feature set of 4×33 dimensions
[0072]
[0073] wherein the superscripts 1-33 represent 33 joint nodes respectively, and (x, y, z, v) represent real-time three-dimensional coordinates and real-time visibility of each joint node respectively. By extracting the real-time action feature set at each time point respectively, the real-time action feature sequence of the moving object object1 corresponding to the time sequence t1, t2, …, t i ,…,t n can be obtained.
[0074] In some preferred embodiments, the real-time action feature extraction unit further includes an angle extraction module configured to determine a real-time deflection angle of each joint node based on the real-time three-dimensional coordinates of the multiple joint nodes, and the real-time action feature set at each time point further includes the real-time deflection angle of each joint node of the moving object at the time point. If Euler rotation information is added to the spatial three-dimensional coordinates of any joint node, the information characterizing the posture of the moving object will be greatly expanded, which can effectively improve the accuracy and generalization ability of action recognition.
[0075] In some preferred embodiments, the real-time action feature extraction unit further comprises a posture correction module configured to correct the real-time action feature sequence based on the real-time three-dimensional coordinates of the plurality of body joints.
[0076] In embodiments of the present application, the standard action feature construction unit is configured to construct a standard action feature sequence of at least one type of exercise and save it in a database. Those skilled in the art will readily know that the standard action feature sequence has the same data format as the real-time action feature sequence, so as to facilitate subsequent identification and evaluation of the real-time action feature sequence as a standard, i.e., for a type of exercise, its standard action feature sequence includes a set of standard action features at a plurality of time points, wherein each set of standard action features at a time point includes standard three-dimensional coordinates of each joint node at the time point and standard visibility. For example, for a set of time series t1, t2, …, t i ,…,t n , the corresponding standard action feature sequence is The specific expression of the standard action feature sequence is:
[0077]
[0078] wherein the meanings of the subscripts are the same as , and (X, Y, Z, V) respectively represent the three-dimensional coordinates that each joint node should be in when performing the type of exercise and the standard visibility that each joint node should have under the shooting angle of the acquisition unit.
[0079] A more ideal way to construct a standard action feature sequence is to have professional practitioners of each type of exercise (such as yoga and other exercises with high requirements on actions) as exercise objects, and to collect and extract real-time action feature sequences of the type of exercise through the acquisition unit and the real-time action extraction unit, and to use them as the standard action feature sequence of the type of exercise. However, in actual implementation, the above implementation will greatly increase the cost of constructing the standard action feature sequence. In comparison, it is undoubtedly more convenient and cost-saving to construct the standard action feature sequence using standard action materials obtained through the Internet and other channels.
[0080] Since the above-mentioned standard action materials are generally two-dimensional videos shot at a specific angle, the standard three-dimensional positions of each joint node are further derived from the two-dimensional information of the standard action video. For this purpose, in embodiments of the present application, the two-dimensional video embodying the standard action of various types of exercise is fused with the real-time three-dimensional position information of the exercise object obtained by the acquisition unit to construct the standard action feature sequence of various types of exercise.
[0081] Specifically, the standard action feature construction unit constructs the standard action feature sequence of each movement type based on the following steps:
[0082] A100: Acquire the standard action video of the movement type;
[0083] A200: Mark a plurality of reference joint nodes from the standard action video of the movement type;
[0084] A300: Select a real-time action feature sequence with the same shooting angle as the standard action video from each real-time action feature sequence as a candidate action feature sequence;
[0085] A400: Adjust the positions of the corresponding joint nodes in the candidate action feature sequence based on the positions of the plurality of reference joint nodes;
[0086] A500: Adjust the positions of the non-corresponding joint nodes in the candidate action feature sequence based on the positions of the corresponding joint nodes in the candidate action feature sequence;
[0087] A600: Generate the standard action feature sequence of the movement type based on the positions of the corresponding joint nodes and the positions of the non-corresponding joint nodes, the standard action feature sequence including a standard action feature set of the movement type at each time point, wherein each standard action feature set at a time point includes the standard three-dimensional coordinates and the standard visibility of each joint node at the time point.
[0088] In some preferred embodiments, the standard visibility is determined based on the plurality of reference joint nodes marked from the standard action video.
[0089] In the above steps, first, the standard action video of a specific movement type is acquired, and the marking of a plurality of reference joint nodes is performed thereon. Obviously, for a standard action video in two-dimensional format, the marked joint nodes are all visible. Then, a real-time action feature sequence with the same shooting angle as the standard action video is selected from each real-time action feature sequence extracted by the real-time action feature extraction unit as a candidate action feature sequence. Then, the positions of the corresponding joint nodes in the candidate action feature sequence are adjusted using the marked reference joint nodes so as to match the reference joint nodes, and the visibility of the corresponding joint nodes is set as visible. Finally, the positions of the non-corresponding joint nodes are adjusted based on the spatial positions, angles, and other limiting conditions (for example, the three-dimensional distance between two adjacent joint nodes should be kept unchanged) between the joint nodes, and the visibility of the non-corresponding joint nodes is set as invisible. The three-dimensional positions and the visibility of all joint nodes are finally determined. The above operations are performed for each time point, and the standard action feature sequence is obtained.
[0090] By using the steps A100-A600, the motion feature sequence of a non-standard degree can be corrected by using a two-dimensional standard motion video which is easy to obtain, so as to obtain a three-dimensional standard motion feature sequence. Figure 6 A diagram showing the standard three-dimensional coordinates of each joint constructed by the standard motion feature construction module in one specific embodiment is shown.
[0091] In the embodiments of the present application, the evaluation unit identifies the motion type and motion effect of the at least one motion object based on the real-time motion feature sequence and the standard motion feature sequence.
[0092] In some preferred embodiments, the evaluation unit can use a trained deep learning network to identify the motion type of the at least one motion object, and specifically, the evaluation unit identifies the motion type and motion effect of the at least one motion object based on the following steps:
[0093] B100: extracting a real-time motion feature sequence of the at least one motion object;
[0094] B200: identifying the motion type of the motion object based on the trained deep learning network;
[0095] B300: extracting a standard motion feature sequence corresponding to the identified motion type;
[0096] B400: correcting the standard visibility of each joint in the standard motion feature sequence corresponding to the identified motion type based on the posture of the real-time motion feature sequence of the motion object;
[0097] B500: counting the number of different real-time visibilities of each joint in the real-time motion feature sequence of the motion object and the standard visibilities of each joint in the corrected standard motion feature sequence, if the counting result is greater than a preset first threshold, the motion effect is evaluated as poor and the step B100 is returned, otherwise, the step B600 is executed;
[0098] B600: calculating the matching degree of the real-time motion feature sequence of the motion object and the corrected standard motion feature sequence, if the matching degree is less than a preset second threshold, the motion effect is evaluated as general and the step B100 is returned, otherwise, the step B700 is executed;
[0099] B700: evaluating the motion effect as good and returning to the step B100.
[0100] In the above steps, firstly, the motion type of the action performed by the motion object is identified through steps B100-B200, and then the motion effect is evaluated through steps B300-B700. Since the angle at which the collection unit captures the motion object when performing various types of motion can be different from the angle at which the standard action is captured, the angle of the standard action feature sequence can be transformed through step B400 to be the same as the angle at which the real-time action feature sequence is captured, and on this basis, the standard visibility of each joint node in the standard action feature sequence is re-corrected according to the occlusion relationship of the human body tissue. For example, in a specific embodiment, the angle at which the standard action feature sequence of performing sit-ups is captured is the side direction, and the real-time action feature sequence is collected by the motion object facing away from the collection unit, so the angle at which the standard action feature sequence is captured should be adjusted based on the collection direction of the real-time action feature sequence, and the standard visibility of the joint node occluded by the human body tissue at this angle is corrected to be invisible.
[0101] After the above operation is completed, the motion effect of the motion object can be evaluated using the corrected standard action feature sequence. In the evaluation process of the motion effect, firstly, the number of different real-time visibility of each joint node in the real-time action feature sequence and the standard visibility of each joint node in the standard action feature sequence is counted through step B500. The deviation of the visibility of each joint node reflects the large deviation between the real-time action of the motion object and the standard action. For example, when the motion object performs this type of motion (such as yoga, sit-ups, push-ups, jumping rope, etc.), the position and rotation angle of each joint node of the body deviates from the standard action, which can cause the number of joint nodes with different real-time visibility and standard visibility to exceed the first threshold value. At this time, the motion effect can be directly evaluated as poor.
[0102] If the number of joint nodes with different real-time visibility and standard visibility is less than or equal to the first threshold value, the matching degree between the real-time action feature sequence and the standard action feature sequence is further evaluated through step B600. For example, the real-time three-dimensional coordinates and standard three-dimensional coordinates of each joint node can be first normalized using the connection line between the left and right shoulder midpoints and the left and right hip midpoints, and then the average value of the Euclidean distance between the real-time three-dimensional coordinates and the standard three-dimensional coordinates of each joint node is taken as the matching degree (or the average value of the difference between the real-time deflection angle of each joint node and the standard deflection angle determined based on the calibrated three-dimensional coordinates can be used as the matching degree). If the matching degree is greater than the second threshold value, the motion effect is evaluated as general, otherwise it can be considered that the real-time action of the motion object is relatively standard, and at this time the motion effect can be evaluated as good.
[0103] In some preferred embodiments, the different positions of the joints, such as the joints of the trunk and the joints of the limbs, can be further classified and counted, and the deviations of the movements of different parts of the body from the standard movements can be further determined according to the counting results, and the corresponding movement accuracy scores, movement guidance or suggestions, etc. can be given. Figure 7a 、 Figure 7b The schematic diagrams respectively showing that one specific moving object has different postures when performing the same type of movement are shown in FIGS. 1A-1C. As shown in FIG. 1A, Figure 7a 、 Figure 7b FIG. 1C, the moving object has relatively standard movements of the limbs and the trunk when performing the same type of movement, and only the angle of the joint of the face deviates from the standard movement, and at this time, the guidance or suggestions of the movement for the specific part can be proposed.
[0104] In the embodiments of the present application, the display unit can be a liquid crystal display, a mobile phone, a tablet computer or the like, which is used to display the real-time three-dimensional position information, the movement type and the movement effect.
[0105] The specific embodiments of the present application are described in detail above, and for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also belong to the protection scope of the claims of the present application.
Claims
1. A smart home fitness assistance system, comprising a data acquisition unit, a real-time motion feature extraction unit, a standard motion feature construction unit, an evaluation unit, a database, and a display unit, characterized in that: The acquisition unit is used to acquire the real-time three-dimensional position information of at least one moving object; The real-time motion feature extraction unit extracts a real-time motion feature sequence of at least one moving object based on the real-time three-dimensional position information; The standard motion feature construction unit is used to construct a standard motion feature sequence of at least one type of motion and store it in the database; The evaluation unit identifies the motion type and motion effect of at least one moving object based on the real-time motion feature sequence and the standard motion feature sequence. The display unit is used to display the real-time three-dimensional position information, motion type, and motion effect; The real-time motion feature extraction unit includes a coordinate extraction module, a people recognition module, a multi-person segmentation module, and a processing module; The real-time motion feature extraction unit extracts a real-time motion feature sequence of at least one moving object based on the following steps: S100: The coordinate extraction module determines the real-time 3D coordinates and real-time visibility of multiple joints based on the real-time 3D position information; S200: The people recognition module identifies the number of people in a moving object based on the real-time three-dimensional coordinates of the multiple joints; S300: If the number of people involved in the movement is greater than 1, proceed to step S400; otherwise, proceed to step S500. S400: The multi-person segmentation module segments the multiple joints based on the number of people in the moving object, and performs step S500 for each segmented moving object; S5 00: The processing module generates a real-time motion feature sequence for each moving object. The real-time motion feature sequence includes a set of real-time motion features of the moving object at multiple time points. The set of real-time motion features at each time point includes the real-time three-dimensional coordinates and real-time visibility of each joint of the moving object at that time point. The standard motion feature construction unit constructs a standard motion feature sequence for each type of motion based on the following steps: A100: Obtain standard motion videos for this type of sport; A200: Identify multiple reference joints from standard motion videos of this type of sport; A300: Select a real-time motion feature sequence from each real-time motion feature sequence that has the same shooting angle as the standard motion video as a candidate motion feature sequence; A400: Adjust the position of the corresponding joint in the candidate action feature sequence based on the position of the plurality of reference joints; A500: Adjust the position of non-corresponding joints in the candidate action feature sequence based on the position of the corresponding joint in the candidate action feature sequence; A600: Generate a standard motion feature sequence for this type of motion based on the positions of the corresponding joints and the positions of the non-corresponding joints. The standard motion feature sequence includes a set of standard motion features for this type of motion at multiple time points, wherein the set of standard motion features at each time point includes the standard three-dimensional coordinates and standard visibility of each joint at that time point.
2. The home fitness intelligent auxiliary system according to claim 1, characterized in that: The multiple joints include multiple facial joints, multiple trunk joints, and multiple limb joints.
3. The home fitness intelligent auxiliary system according to claim 2, characterized in that, The people recognition module uses at least one of the following criteria to determine the number of people in the moving object: The number of faces is determined based on the number of facial joints, the total number of joints, and the three-dimensional distance between any two joints.
4. The home fitness intelligent auxiliary system according to claim 1, characterized in that: The real-time motion feature extraction unit further includes an angle extraction module, which determines the real-time deflection angle of each joint point based on the real-time three-dimensional coordinates of the multiple joint points, and... The real-time motion feature set at each time point also includes the real-time deflection angle of each joint of the moving object at that time point.
5. The home fitness intelligent auxiliary system according to claim 2, characterized in that: The real-time motion feature extraction unit further includes a posture correction module, which performs posture correction on the real-time motion feature sequence based on the real-time three-dimensional coordinates of the multiple trunk joints.
6. The home fitness intelligent auxiliary system according to claim 1, characterized in that: The standard visibility is determined based on multiple reference joints calibrated from the standard motion video.
7. The home fitness intelligent auxiliary system according to claim 1, characterized in that: The evaluation unit uses a trained deep learning network to identify the motion type of at least one moving object.
8. The home fitness intelligent auxiliary system according to claim 7, characterized in that, The evaluation unit identifies the movement type and movement effect of at least one moving object based on the following steps: B100: Extract the real-time motion feature sequence of at least one moving object; B200: Identify the motion type of the moving object based on the trained deep learning network; B300: Extract the standard motion feature sequence corresponding to the identified motion type; B400: Based on the pose of the real-time motion feature sequence of the moving object, the standard visibility of each joint in the standard motion feature sequence corresponding to the identified motion type is corrected; B500: Count the number of times the real-time visibility of each joint in the real-time motion feature sequence of the moving object is different from the standard visibility of each joint in the corrected standard motion feature sequence. If the statistical result is greater than the preset first threshold, the motion effect is evaluated as poor and the process returns to step B100; otherwise, step B600 is executed. B600: Calculate the matching degree between the real-time motion feature sequence of the moving object and the corrected standard motion feature sequence. If the matching degree is less than the preset second threshold, the motion effect is evaluated as average and the process returns to step B100; otherwise, step B700 is executed. B700: Assess the exercise effect as good and return to step B100.