Action recognition method and device, equipment and storage medium
By creating an action database and analyzing video data using neural network algorithm models, the problem of inaccurate recognition of free force movements in two-dimensional video recognition technology is solved, and accurate identification and effective statistics of the number of movements are achieved.
Patent Information
- Application Number
- CN202410772922.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-14
- Publication Date
- 2025-02-18
AI Technical Summary
In two-dimensional video recognition technology, object detection technology based on YOLO model will cause inaccurate counting times when identifying the number of free force movements, especially when the instrument moves from near to far, resulting in abnormal movements being counted. times, and then error data is generated, and the number of exercises cannot be counted effectively.
By creating a start action database and an end action database, the video data is analyzed frame by frame using neural network algorithm model, the position information of the moving equipment is obtained, and the action start and end are determined based on the longitudinal displacement changes, so as to accurately identify the number of movements.
The accurate identification of the number of exercises is achieved, the problem of increasing the number of exercises in abnormal movements is avoided, and the accuracy of the number of exercises is ensured.
Smart Images
Figure CN119479053A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of recognition technology, and in particular to an action recognition method, device, equipment and storage medium. Background Art
[0002] Under the two-dimensional video recognition technology, object detection technology based on the YOLO (You Only Look Once, a deep learning model for target detection) model will result in inaccurate counting when identifying the number of free force exercises. The movement calculation deviation of the equipment from near to far will result in movement counting in abnormal movements, which will generate erroneous data and make it impossible to effectively count the number of exercises. Therefore, how to determine whether the number of machine movements is within the effective range of action and effectively count the number of movements is a problem that needs to be solved urgently. Summary of the invention
[0003] In view of this, the purpose of the present invention is to provide a motion recognition method, device, equipment and storage medium, which can solve the problem of the increase in the number of movements of users under abnormal motion conditions and accurately recognize the number of movements. The specific scheme is as follows:
[0004] In a first aspect, the present application discloses an action recognition method, comprising:
[0005] Create a start action database and an end action database;
[0006] Analyze the video data of the user completing the target movement frame by frame using a neural network algorithm model to obtain the position information of the sports equipment in each frame, and determine the longitudinal displacement change of the sports equipment in the current frame and the sports equipment in the previous frame based on the position information;
[0007] When the value corresponding to the longitudinal displacement change is not equal to zero, generating first dimensional data based on the target joint point information of the user in the picture corresponding to the current frame, comparing the first dimensional data with the file data in the start action database, and determining whether the current action of the user is a motion start action according to the corresponding first comparison result;
[0008] When the value corresponding to the longitudinal displacement change is equal to zero, generating second dimensional data based on the target joint point information of the user in the picture corresponding to the current frame, comparing the second dimensional data with the file data in the end action database, and determining whether the current action of the user is a movement end action according to the corresponding second comparison result;
[0009] The number of effective free-strength exercises of the user is determined according to the first comparison result and the second comparison result.
[0010] Optionally, the creating of the start action database and the end action database includes:
[0011] Taking a number of start and end pictures of free strength exercise, and dividing the start and end pictures into two groups for saving, so as to obtain a start action picture folder and an end action picture folder;
[0012] Processing each of the start and end pictures to obtain the target joint point information of the user in the start and end pictures;
[0013] Convert each position coordinate value corresponding to the target joint point information to obtain corresponding multi-dimensional coordinate data;
[0014] The multi-dimensional coordinate data corresponding to each of the pictures are saved respectively to the start action picture folder and the end action picture folder according to the action type, so as to determine the start action database and the end action database based on the start action picture folder and the end action picture folder; the action type includes a start action type and an end action type.
[0015] Optionally, comparing the first dimensional data with the file data in the start action database, and determining whether the current action of the user is a motion start action according to a corresponding first comparison result, includes:
[0016] Acquire all file data from the start action database, subtract the first dimension data from the file data in sequence to acquire corresponding absolute values of first differences, and determine first array data based on the absolute values of the first differences;
[0017] Obtain a group of first arrays with the largest absolute values from the first array data, and obtain a first number of error data with the smallest absolute values from each first array corresponding to the file data, so as to determine a first target dimension array;
[0018] Calculate the mean of each data in the first target dimension array to obtain a first mean array;
[0019] Bubble sort is performed on the first mean array to obtain a second number of data with the smallest mean, and the second number of data is grouped according to the action names of free strength exercises, and it is determined whether the user's current action is an exercise start action based on the size relationship between the corresponding first grouping result and a preset threshold parameter.
[0020] Optionally, determining whether the current action of the user is a motion start action according to a magnitude relationship between the corresponding first grouping result and a preset threshold parameter includes:
[0021] If the number of start actions of the target movement in the free power movement in the first grouping result is greater than or equal to the preset threshold parameter, determining that the current action of the user is the movement start action of the target movement of the free power movement;
[0022] If the number of starting actions of the target movement in the free power movement in the first grouping result is less than the preset threshold parameter, it is determined that the current action of the user is an invalid action.
[0023] Optionally, comparing the second dimensional data with the file data in the end action database, and determining whether the current action of the user is a movement end action according to a corresponding second comparison result, includes:
[0024] Acquire all file data from the end action database, subtract the second dimension data from the file data in sequence to acquire the absolute value of the corresponding second difference, and determine the second array data based on the absolute value of the second difference;
[0025] Obtain a group of second arrays with the largest absolute value from the second array data, and obtain a first number of error data with the smallest absolute value from each second array corresponding to the file data, so as to determine a second target dimension array;
[0026] Calculate the mean of each data in the second target dimension array to obtain a second mean array;
[0027] Bubble sort is performed on the second mean array to obtain a second number of data with the smallest mean, and the second number of data is grouped according to the action names of free strength exercises, and whether the user's current action is an exercise ending action is determined based on the size relationship between the corresponding second grouping result and a preset threshold parameter.
[0028] Optionally, determining whether the current action of the user is a movement ending action according to a magnitude relationship between the corresponding second grouping result and a preset threshold parameter includes:
[0029] If the number of the ending actions of the target movement in the free power movement in the second grouping result is greater than or equal to the preset threshold parameter, determining that the current action of the user is the movement ending action of the target movement of the free power movement;
[0030] If the number of the ending actions of the target movement in the free power movement in the second grouping result is less than the preset threshold parameter, it is determined that the current action of the user is an invalid action.
[0031] Optionally, determining the number of effective free-strength exercises of the user according to the first comparison result and the second comparison result includes:
[0032] If the actions corresponding to the movement start action and the movement end action are judged to be the start or end of the same movement action, then the current free power exercise of the user is determined to be a valid action data, so as to determine the number of effective free power exercises of the user based on the judgment of the valid action data.
[0033] In a second aspect, the present application discloses a motion recognition device, comprising:
[0034] A database creation module, used to create a start action database and an end action database;
[0035] A position information acquisition module is used to analyze the video data of the user completing the target movement frame by frame using a neural network algorithm model to obtain the position information of the sports equipment in each frame;
[0036] a longitudinal displacement change determination module, used to determine the longitudinal displacement change of the sports equipment in the current frame and the sports equipment in the previous frame based on the position information;
[0037] a first comparison module, configured to generate first dimensional data based on the target joint point information of the user in the picture corresponding to the current frame when the value corresponding to the longitudinal displacement change is not equal to zero, compare the first dimensional data with the file data in the start action database, and determine whether the current action of the user is a motion start action according to the corresponding first comparison result;
[0038] A second comparison module is used to generate second dimensional data based on the target joint point information of the user in the picture corresponding to the current frame when the value corresponding to the longitudinal displacement change is equal to zero, compare the second dimensional data with the file data in the end action database, and determine whether the current action of the user is a movement end action according to the corresponding second comparison result;
[0039] The exercise number determination module is used to determine the number of effective free-strength exercises of the user according to the first comparison result and the second comparison result.
[0040] In a third aspect, the present application discloses an electronic device, comprising:
[0041] Memory, used to store computer programs;
[0042] A processor is used to execute the computer program to implement the aforementioned action recognition method.
[0043] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, which implements the aforementioned action recognition method when executed by a processor.
[0044] When performing action recognition, the present application first creates a start action database and an end action database, and uses a neural network algorithm model to perform frame-by-frame analysis on the video data of the user completing the target movement, so as to obtain the position information of the sports equipment in each frame, and determine the longitudinal displacement change of the sports equipment in the current frame and the sports equipment in the previous frame based on the position information; then when the value corresponding to the longitudinal displacement change is not equal to zero, first dimensional data is generated based on the target joint point information of the user in the picture corresponding to the current frame, the first dimensional data is compared with the file data in the start action database, and whether the current action of the user is a movement start action is determined according to the corresponding first comparison result; thereafter, when the value corresponding to the longitudinal displacement change is equal to zero, second dimensional data is generated based on the target joint point information of the user in the picture corresponding to the current frame, the second dimensional data is compared with the file data in the end action database, and whether the current action of the user is a movement end action is determined according to the corresponding second comparison result; finally, the number of effective free force movements of the user is determined according to the first comparison result and the second comparison result. It can be seen that this application uses the action parameters as the reference system parameters of the device displacement to make a three-dimensional judgment on the two-dimensional video, and determine whether the number of machine movements is within the effective action range, thereby solving the problem of the user's increased number of movements under abnormal action conditions. At the same time, the device actively identifies the user's actions, and the user does not need to manually enter the action information, so the effective number of free force exercises can be identified more efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0046] Figure 1 A flow chart of an action recognition method disclosed in this application;
[0047] Figure 2 A schematic diagram of location information disclosed in this application;
[0048] Figure 3 This is a schematic diagram of the structure of a motion recognition device disclosed in this application;
[0049] Figure 4This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] At present, the technology of moving object detection using deep learning algorithms, such as object detection technology based on the YOLO model, cannot determine the displacement of two-dimensional videos in three-dimensional space. The existing technology lacks three-dimensional spatial displacement determination and cannot effectively count the number of movements. In order to solve the above technical problems, the present application discloses an action recognition method that can solve the problem of the increase in the number of movements of users under abnormal action conditions and accurately recognize the number of movements.
[0052] See also Figure 1 As shown, an embodiment of the present invention discloses a motion recognition method, including:
[0053] Step S11: Create a start action database and an end action database.
[0054] In this embodiment, firstly, a number of start and end pictures of free strength exercise are taken, and the start and end pictures are divided into two groups for saving, so as to obtain a start action picture folder and an end action picture folder; each start and end picture is processed to obtain the target joint point information of the user in the start and end pictures; each position coordinate value corresponding to the target joint point information is converted to obtain the corresponding multi-dimensional coordinate data; the multi-dimensional coordinate data corresponding to each picture is saved to the start action picture folder and the end action picture folder according to the action type, so as to determine the start action database and the end action database based on the start action picture folder and the end action picture folder; the action type includes the start action type and the end action type. In detail, a large number of start and end pictures of barbell squat exercise are taken, and all pictures are divided into two groups, which are saved to the start action picture folder and the end action picture folder respectively; then each picture is processed by using a neural network algorithm model to obtain the positions of multiple joint points of the athlete in the picture. As an embodiment, the number of joint points a can be 32, and the number of joint points in this application is 17≤a≤32. The following is a specific explanation with a=32. The joints include left hip (left_hip), right hip (right_hip), left shoulder (left_shoulder), right shoulder (right_shoulder), left elbow (left_elbow), right elbow (right_elbow), left wrist (left_wrist), right wrist (right_wrist), left heel (left_heel), right heel (right_heel), left knee (left_knee), right knee (right_knee), left ankle (left_ankle), right ankle (right_ankle), left foot (left_foot_index), right foot (right_foot_index), left thumb (left_thumb), right thumb (right_thumb). ght_thumb), left index finger (left_index), right index finger (right_index), left pinky (left_pinky), right pinky (right_pinky), left corner of mouth (mouth_left), right corner of mouth (mouth_right), left ear (left_ear), right ear (right_ear), left inner corner of eye (left_eye_inner), left eye (left_eye), left outer corner of eye (left_eye_outer), right inner corner of eye (right_eye_inner), right eye (right_eye), right outer corner of right eye (right_eye_outer); convert the position coordinate values of the above 32 joint points to generate a multidimensional space coordinate object. The multidimensional space coordinate algorithm is as follows:
[0055] Dimension 1: X1 is the torso distance, which is the mean height of the left hip (left_hip) and the right hip (right_hip) minus the mean height of the left shoulder (left_shoulder) and the right shoulder (right_shoulder);
[0056] The second dimension: X2 is the distance between the left shoulder (left_shoulder) and the left elbow (left_elbow);
[0057] The third dimension: X3 is the distance between the right shoulder (right_shoulder) and the right elbow (right_elbow);
[0058] The fourth dimension: X4 is the distance between the left elbow (left_elbow) and the left wrist (left_wrist);
[0059] The fifth dimension: X5 is the distance between the right elbow (right_elbow) and the right wrist (right_wrist);
[0060] Dimension 6: X6 is the distance between the left hip (left_hip) and the left knee (left_knee);
[0061] Dimension 7: X7 is the distance between the right hip (right_hip) and the right knee (right_knee);
[0062] 8th dimension: X8 is the distance between the left knee (left_knee) and the left ankle (left_ankle);
[0063] 9th dimension: X9 is the distance between the right knee (right_knee) and the right ankle (right_ankle);
[0064] 10th Dimension: X 10 is the distance between the left shoulder (left_shoulder) and the left wrist (left_wrist);
[0065] 11th Dimension: X 11 is the distance between the right shoulder (right_shoulder) and the right wrist (right_wrist);
[0066] 12th Dimension: X 12 is the distance between the left hip (left_hip) and the left ankle (left_ankle);
[0067] 13th Dimension: X 13 is the distance between the right hip (right_hip) and the right ankle (right_ankle);
[0068] 14th Dimension: X 14 is the distance between the left hip (left_hip) and the left wrist (left_wrist);
[0069] 15th Dimension: X 15 is the distance between the right hip (right_hip) and the right wrist (right_wrist);
[0070] 16th Dimension: X 16 is the distance between the left shoulder (left_shoulder) and the left ankle (left_ankle);
[0071] 17th Dimension: X 17 is the distance between the right shoulder (right_shoulder) and the right ankle (right_ankle);
[0072] 18th Dimension: X 18 is the distance between the left hip (left_hip) and the left wrist (left_wrist);
[0073] 19th Dimension: X 19 is the distance between the right hip (right_hip) and the right wrist (right_wrist);
[0074] 20th Dimension: X 20 is the distance between the left elbow (left_elbow) and the right elbow (right_elbow);
[0075] 21st Dimension: X 21 is the distance between right knee (right_knee) and right knee (right_knee);
[0076] 22nd Dimension: X 22 is the distance between the left wrist (left_wrist) and the right wrist (right_wrist);
[0077] 23rd Dimension: X 23 is the distance between the left ankle (left_ankle) and the right ankle (right_ankle);
[0078] Thus, a 23-dimensional coordinate data is generated according to the positions of the 32 joint points: embedding = (x1, x2, x3, x4, ⋯, x 23); Generate a 23-dimensional data for each picture, and then save it into two csv (Comma-Separated Values) files according to the starting and ending actions, marked as squat_start.csv and squat_finish.csv, each file has multiple embedding data; refer to the above process to generate corresponding csv files for the starting and ending actions of other free strength exercises such as barbell bench press and barbell deadlift; save the csv files generated by the starting actions of all exercises into the starting action folder, and save the csv files generated by the ending actions of all exercises into the ending action folder to generate the action basic database, that is, generate the starting action database and the ending action database containing all exercise actions.
[0079] Step S12: Analyze the video data of the user completing the target movement frame by frame using a neural network algorithm model to obtain the position information of the sports equipment in each frame, and determine the longitudinal displacement change of the sports equipment in the current frame and the sports equipment in the previous frame based on the position information.
[0080] In the recognition process, the camera is first used to obtain the video stream data of the user completing the barbell squat exercise, and the video data is transmitted to the processor; the processor uses the neural network algorithm model to analyze the video frame by frame to obtain the position information of each frame, such as Figure 2 As shown in the figure, taking a barbell as an example, the barbell position information is position=(x, y, w, h); where x is the vertical distance from the upper left corner of the barbell in the current image to the leftmost edge of the current image, y is the vertical distance from the upper left corner of the barbell to the top of the current image, w is the length of the entire barbell in the current image, and h is the height of the entire barbell in the current image. After that, the longitudinal displacement change between the current frame n and the previous frame n-1 is: ∆y=y n -y n-1 ;y n y is the vertical distance from the upper left corner of the barbell to the top of the current image in the current frame; n-1 It is the vertical distance from the upper left corner of the barbell in the previous frame to the top of the current image.
[0081] Step S13: When the value corresponding to the longitudinal displacement change is not equal to zero, generate first dimensional data based on the target joint point information of the user in the corresponding picture of the current frame, compare the first dimensional data with the file data in the start action database, and determine whether the current action of the user is a motion start action based on the corresponding first comparison result.
[0082] In this embodiment, when ∆y=0 changes to ∆y≠0, in a specific embodiment, it means that the barbell starts to move in the vertical position. Whether it is the start of the barbell squat exercise or the horizontal dragging of the barbell, a vertical visual displacement effect will be generated in the video stream. Generate a 23D data embedding E from the current frame data x =(x1, x2, x3, x4, ⋯, x 23 ).
[0083] Specifically, when generating a 23-dimensional data from the current frame data, the neural network algorithm model is first used to process each image to obtain the positions of the 32 joint points of the athlete in the image, and the position coordinate values of the 32 joint points are converted to generate a multi-dimensional space coordinate object. The multi-dimensional space coordinate algorithm is shown in step S11. Finally, a 23-dimensional coordinate data is generated according to the positions of the 32 joint points as embedding=(x1, x2, x3, x4, ⋯, x 23 ); that is, at this time, first dimension data is generated based on the target joint point information of the user in the picture corresponding to the current frame, the first dimension data is compared with the file data in the start action database, and whether the current action of the user is a motion start action is determined according to the corresponding first comparison result. When making an action judgment, all file data are obtained from the start action database, the first dimension data and the file data are subtracted in sequence to obtain the absolute value of the corresponding first difference, and the first array data is determined based on the absolute value of the first difference; a group of first arrays with the largest absolute value are obtained from the first array data, and a first number of error data with the smallest absolute value are obtained from each first array corresponding to the file data to determine the first target dimension array; each data in the first target dimension array is averaged to obtain a first mean array; the first mean array is bubble sorted to obtain a second number of data with the smallest mean, and the second number of data is grouped according to the action name of the free power exercise, and whether the current action of the user is a motion start action is determined according to the size relationship between the corresponding first grouping result and the preset threshold parameter. If the number of starting actions of the target movement in the free power exercise in the first grouping result is greater than or equal to the preset threshold parameter, the user's current action is determined to be the starting action of the target movement of the free power exercise; if the number of starting actions of the target movement in the free power exercise in the first grouping result is less than the preset threshold parameter, the user's current action is determined to be an invalid action.
[0084] Step S14: When the value corresponding to the longitudinal displacement change is equal to zero, generate second dimensional data based on the target joint point information of the user in the corresponding picture of the current frame, compare the second dimensional data with the file data in the end action database, and determine whether the current action of the user is an end-of-motion action based on the corresponding second comparison result.
[0085] In another specific embodiment, when ∆y≠0 changes to ∆y=0, it means that the barbell stops moving. No matter when the barbell squat ends or the horizontal dragging of the barbell ends, a vertical visual barbell stops in the video stream. At this time, the second dimension data is generated based on the target joint point information of the user in the picture corresponding to the current frame, that is, the current frame data is generated into a 23-dimensional data embedding E x =(x1, x2, x3, x4, ⋯, x 23 ), the specific 23 data generation process is consistent with the process in step S11, and will not be repeated here. After obtaining the second dimension data, by inputting the current frame data and all csv files in the end action library, it can be determined that the action performed by the current frame data is the end action or invalid action of the barbell squat. Specifically, all file data are obtained from the end action database, the second dimension data and the file data are subtracted in turn to obtain the absolute value of the corresponding second difference, and the second array data is determined based on the absolute value of the second difference; a group of second arrays with the largest absolute value are obtained from the second array data, and the first number of error data with the smallest absolute value is obtained from each second array corresponding to the file data to determine the second target dimension array; each data in the second target dimension array is averaged to obtain a second mean array; the second mean array is bubble sorted to obtain the second number of data with the smallest mean, and the second number of data is grouped according to the action name of the free power exercise, and the user's current action is determined to be the end action of the exercise according to the size relationship between the corresponding second grouping result and the preset threshold parameter. If the number of ending actions of the target movement in the free power exercise in the second grouping result is greater than or equal to the preset threshold parameter, the user's current action is determined to be the movement ending action of the target movement in the free power exercise; if the number of ending actions of the target movement in the free power exercise in the second grouping result is less than the preset threshold parameter, the user's current action is determined to be an invalid action.
[0086] Step S15: Determine the number of effective free-strength exercises of the user according to the first comparison result and the second comparison result.
[0087] In this embodiment, if the action corresponding to the movement start action and the movement end action is judged as the start or end of the same movement action, the free strength exercise of the current user is determined to be a valid action data, so as to determine the number of effective free strength exercises of the user based on the judgment of the effective action data. Taking the barbell as an example, if the action judgment data obtained by the start action and the end action is the start or end of the same action (such as: barbell squat), it can be confirmed that the current movement is a complete action data; if the action judgment data obtained by the start and end actions are not the same action, then the current movement will be regarded as an invalid movement, and no data record will be made; if the action judgment data obtained by the start or end action is an invalid action, then the current movement will also be regarded as an invalid data, and no data record will be made. Finally, the number of effective free strength exercises of the user is determined based on the first comparison result and the second comparison result. It should be pointed out that the displacement difference ∆y of the action (such as: barbell squat) can also be calculated based on the start and end actions of the valid data to complete the calculation of relevant data such as speed, displacement, and work.
[0088] In summary, when performing action recognition, the present application first creates a start action database and an end action database, and uses a neural network algorithm model to perform frame-by-frame analysis on the video data of the user completing the target movement, so as to obtain the position information of the sports equipment in each frame, and determine the longitudinal displacement change of the sports equipment in the current frame and the sports equipment in the previous frame based on the position information; then when the value corresponding to the longitudinal displacement change is not equal to zero, a first dimension data is generated based on the target joint point information of the user in the picture corresponding to the current frame, and the first dimension data is compared with the file data in the start action database, and whether the current action of the user is a movement start action is determined according to the corresponding first comparison result; thereafter, when the value corresponding to the longitudinal displacement change is equal to zero, a second dimension data is generated based on the target joint point information of the user in the picture corresponding to the current frame, and the second dimension data is compared with the file data in the end action database, and whether the current action of the user is a movement end action is determined according to the corresponding second comparison result; finally, the number of effective free force movements of the user is determined according to the first comparison result and the second comparison result. It can be seen that this application uses the action parameters as the reference system parameters of the device displacement to make a three-dimensional judgment on the two-dimensional video, and determine whether the number of machine movements is within the effective action range, thereby solving the problem of the user's increased number of movements under abnormal action conditions. At the same time, the device actively identifies the user's actions, and the user does not need to manually enter the action information, so the effective number of free force exercises can be identified more efficiently and accurately.
[0089] Based on the previous embodiment, it can be known that when judging the current action of the user, the present application generates dimension data based on the target joint point information of the user in the picture corresponding to the current frame, compares the dimension data with the file data in the action database, and determines whether the current action of the user is a motion start action or a motion end action according to the corresponding comparison result. Next, the specific process of judging the current action of the user will be described in detail.
[0090] First, obtain the number of user target joint points a, determine the dimension number b of the generated data based on a, and generate a b-dimensional data embedding E for the current frame data. x =(x1, x2, x3, x4, ⋯, x b ); match the current frame data with the data of all csv files in the start action library. The operation rules are as follows:
[0091] Get all embedding data from each database and generate an array record: S = (E1, E2, E3, E4, ⋯, E n ); In the operation rules, n is the number of data, a is the number of joints, and b is the number of dimensions. b is the number of dimensions obtained by performing a difference operation on a, where 17≤a≤32, b>0, and b is an empirical value. For example, when a=32, b=23, which means that 32 target joints of the user are obtained, and the number of dimensions of the generated data is determined to be 23 according to the number of target joints 32, and the current frame data is generated into a 23-dimensional data embedding E x =(x1, x2, x3, x4, ⋯, x 23 );
[0092] Subtract the current frame embedding from the data in array S in turn, and get the absolute value of the difference ∆E n =(E x -E n )=(∆x1,∆x2,∆x3,∆x4,⋯,∆x b ); Get a new array data ∆S=(∆E1, ∆E2, ∆E3, ∆E4, ⋯, ∆E n );
[0093] From ∆S, each ∆E n b-dimensional data in the dataset, and obtain the dimension ∆x with the largest absolute value n , generate a new one-dimensional array with the absolute value;
[0094] Perform difference calculation on the data in all csv files in the start action library, and obtain c (here c can be 20, 30 or 40, c is the number of error data with the smallest absolute value, and the range of c is 10-50) error data with the smallest absolute value, and generate a new b-dimensional array: ∆S=(∆E1, ∆E2, ∆E3, ∆E4, ⋯, ∆E c );
[0095] Calculate the mean of the b-dimensional difference data in each item of ∆S: =(∆x1+∆x2+∆x3+∆x4+⋯+∆x b )÷b, recorded as a one-dimensional array of the mean of the difference =( , , , ,⋯, );
[0096] right Perform bubble sorting to obtain the first d (for example, d can be 8, 10, or 15, the range of d is 5-20, and d is less than c) pieces of data with the smallest mean value, and group the d pieces of data according to the action name;
[0097] For example, if among 10 data, 6 are barbell squat starting movements, 2 are barbell bench press starting movements, and 2 are barbell deadlift starting movements, then according to the pre-set judgment threshold parameter e=6 (the range of e is 3-10, and e is less than the value of d), it can be determined that the current movement is the starting movement of the barbell squat;
[0098] If among the 10 data, there are 4 starting movements for barbell squats, 4 starting movements for barbell bench presses, and 2 starting movements for barbell deadlifts, then according to the pre-set judgment threshold parameter e=6, it can be determined that the current movement is not in the initial movement library, which means that the current movement is an invalid movement.
[0099] Similarly, the determination of the end of exercise action is the same as the principle of the above steps, which will not be repeated here.
[0100] In this way, the problem of actively identifying user movements through the device is solved, and the user does not need to manually enter the movement information; it also solves the problem of the user's increased number of exercises when using the device under abnormal movement conditions, and the effective number of free strength exercises is identified more efficiently and accurately.
[0101] See also Figure 3 As shown, an embodiment of the present invention discloses a motion recognition device, including:
[0102] A database creation module 11 is used to create a start action database and an end action database;
[0103] The position information acquisition module 12 is used to analyze the video data of the user completing the target movement frame by frame using a neural network algorithm model to obtain the position information of the sports equipment in each frame;
[0104] A longitudinal displacement change determination module 13, used to determine the longitudinal displacement change of the sports equipment in the current frame and the sports equipment in the previous frame based on the position information;
[0105] A first comparison module 14 is used for generating first dimensional data based on the target joint point information of the user in the picture corresponding to the current frame when the value corresponding to the longitudinal displacement change is not equal to zero, comparing the first dimensional data with the file data in the start action database, and determining whether the current action of the user is a motion start action according to the corresponding first comparison result;
[0106] A second comparison module 15 is used to generate second dimensional data based on the target joint point information of the user in the picture corresponding to the current frame when the value corresponding to the longitudinal displacement change is equal to zero, compare the second dimensional data with the file data in the end action database, and determine whether the current action of the user is a movement end action according to the corresponding second comparison result;
[0107] The exercise number determination module 16 is used to determine the number of effective free-strength exercises of the user according to the first comparison result and the second comparison result.
[0108] In summary, when performing action recognition, the present application first creates a start action database and an end action database, and uses a neural network algorithm model to perform frame-by-frame analysis on the video data of the user completing the target movement, so as to obtain the position information of the sports equipment in each frame, and determine the longitudinal displacement change of the sports equipment in the current frame and the sports equipment in the previous frame based on the position information; then when the value corresponding to the longitudinal displacement change is not equal to zero, a first dimension data is generated based on the target joint point information of the user in the picture corresponding to the current frame, and the first dimension data is compared with the file data in the start action database, and whether the current action of the user is a movement start action is determined according to the corresponding first comparison result; thereafter, when the value corresponding to the longitudinal displacement change is equal to zero, a second dimension data is generated based on the target joint point information of the user in the picture corresponding to the current frame, and the second dimension data is compared with the file data in the end action database, and whether the current action of the user is a movement end action is determined according to the corresponding second comparison result; finally, the number of effective free force movements of the user is determined according to the first comparison result and the second comparison result. It can be seen that this application uses the action parameters as the reference system parameters of the device displacement to make a three-dimensional judgment on the two-dimensional video, and determine whether the number of machine movements is within the effective action range, thereby solving the problem of the user's increased number of movements under abnormal action conditions. At the same time, the device actively identifies the user's actions, and the user does not need to manually enter the action information, so the effective number of free force exercises can be identified more efficiently and accurately.
[0109] In some specific embodiments, the database creation module 11 can be specifically used to take a number of start and end pictures of free strength exercises, and divide the start and end pictures into two groups for saving to obtain a start action picture folder and an end action picture folder; process each of the start and end pictures to obtain the target joint point information of the user in the start and end pictures; convert each position coordinate value corresponding to the target joint point information to obtain corresponding multi-dimensional coordinate data; save the multi-dimensional coordinate data corresponding to each of the pictures to the start action picture folder and the end action picture folder according to the action type, so as to determine the start action database and the end action database based on the start action picture folder and the end action picture folder; the action type includes a start action type and an end action type.
[0110] In some specific embodiments, the first comparison module 14 can be specifically used to obtain all file data from the start action database, subtract the first dimension data from the file data in sequence to obtain the absolute value of the corresponding first difference, and determine the first array data based on the absolute value of the first difference; obtain a group of first arrays with the largest absolute value from the first array data, and obtain a first number of error data with the smallest absolute value from each first array corresponding to the file data to determine a first target dimension array; calculate the mean of each data in the first target dimension array to obtain a first mean array; bubble sort the first mean array to obtain a second number of data with the smallest mean, and group the second number of data according to the action name of the free power exercise, and determine whether the user's current action is a movement start action based on the size relationship between the corresponding first grouping result and the preset threshold parameter.
[0111] In some specific embodiments, the first comparison module 14 can be specifically used to determine that the user's current action is the movement starting action of the target movement of the free power movement if the number of starting actions of the target movement in the free power movement in the first grouping result is greater than or equal to the preset threshold parameter; if the number of starting actions of the target movement in the free power movement in the first grouping result is less than the preset threshold parameter, determine that the user's current action is an invalid action.
[0112] In some specific embodiments, the second comparison module 15 can be specifically used to obtain all file data from the end action database, subtract the second dimension data from the file data in sequence to obtain the absolute value of the corresponding second difference, and determine the second array data based on the absolute value of the second difference; obtain a group of second arrays with the largest absolute value from the second array data, and obtain a first number of error data with the smallest absolute value from each second array corresponding to the file data to determine a second target dimension array; calculate the mean of each data in the second target dimension array to obtain a second mean array; bubble sort the second mean array to obtain a second number of data with the smallest mean, and group the second number of data according to the action name of the free strength exercise, and determine whether the user's current action is an exercise end action based on the size relationship between the corresponding second grouping result and the preset threshold parameter.
[0113] In some specific embodiments, the second comparison module 15 can be specifically used to determine that the user's current action is the movement ending action of the target movement of the free power movement if the number of ending actions of the target movement in the free power movement in the second grouping result is greater than or equal to the preset threshold parameter; if the number of ending actions of the target movement in the free power movement in the second grouping result is less than the preset threshold parameter, determine that the user's current action is an invalid action.
[0114] In some specific embodiments, the exercise number determination module 16 can be specifically used to determine that the current free strength exercise of the user is a valid action data if the action corresponding to the exercise start action and the exercise end action is judged to be the start or end of the same exercise action, so as to determine the number of effective free strength exercises of the user based on the judgment of the valid action data.
[0115] Furthermore, the present application also discloses an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.
[0116] Figure 4 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the action recognition method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0117] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0118] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.
[0119] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the motion recognition method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.
[0120] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed motion recognition method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.
[0121] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0122] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0123] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0124] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0125] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for motion recognition, characterized in that: include: Create a start action database and an end action database; Analyze the video data of the user completing the target movement frame by frame using a neural network algorithm model to obtain the position information of the sports equipment in each frame, and determine the longitudinal displacement change of the sports equipment in the current frame and the sports equipment in the previous frame based on the position information; When the value corresponding to the longitudinal displacement change is not equal to zero, generating first dimensional data based on the target joint point information of the user in the picture corresponding to the current frame, comparing the first dimensional data with the file data in the start action database, and determining whether the current action of the user is a motion start action according to the corresponding first comparison result; When the value corresponding to the longitudinal displacement change is equal to zero, generating second dimensional data based on the target joint point information of the user in the picture corresponding to the current frame, comparing the second dimensional data with the file data in the end action database, and determining whether the current action of the user is a movement end action according to the corresponding second comparison result; The number of effective free-strength exercises of the user is determined according to the first comparison result and the second comparison result.
2. The action recognition method according to claim 1, characterized in that: The step of creating a start action database and an end action database includes: Taking a number of start and end pictures of free strength exercise, and dividing the start and end pictures into two groups for saving, so as to obtain a start action picture folder and an end action picture folder; Processing each of the start and end pictures to obtain the target joint point information of the user in the start and end pictures; Convert each position coordinate value corresponding to the target joint point information to obtain corresponding multi-dimensional coordinate data; The multi-dimensional coordinate data corresponding to each of the pictures are saved respectively to the start action picture folder and the end action picture folder according to the action type, so as to determine the start action database and the end action database based on the start action picture folder and the end action picture folder; the action type includes a start action type and an end action type.
3. The action recognition method according to claim 1, characterized in that: The comparing the first dimension data with the file data in the start action database, and determining whether the current action of the user is a motion start action according to the corresponding first comparison result, includes: Acquire all file data from the start action database, subtract the first dimension data from the file data in sequence to acquire corresponding absolute values of first differences, and determine first array data based on the absolute values of the first differences; Obtain a group of first arrays with the largest absolute values from the first array data, and obtain a first number of error data with the smallest absolute values from each first array corresponding to the file data, so as to determine a first target dimension array; Calculate the mean of each data in the first target dimension array to obtain a first mean array; Bubble sort is performed on the first mean array to obtain a second number of data with the smallest mean, and the second number of data is grouped according to the action names of free strength exercises, and it is determined whether the user's current action is an exercise start action based on the size relationship between the corresponding first grouping result and a preset threshold parameter.
4. The action recognition method according to claim 3, characterized in that: The determining whether the current action of the user is a motion start action according to the size relationship between the corresponding first grouping result and a preset threshold parameter includes: If the number of start actions of the target movement in the free power movement in the first grouping result is greater than or equal to the preset threshold parameter, determining that the current action of the user is the movement start action of the target movement of the free power movement; If the number of starting actions of the target movement in the free power movement in the first grouping result is less than the preset threshold parameter, it is determined that the current action of the user is an invalid action.
5. The action recognition method according to claim 1, characterized in that: The comparing the second dimensional data with the file data in the end action database, and determining whether the current action of the user is a motion end action according to the corresponding second comparison result, includes: Acquire all file data from the end action database, subtract the second dimension data from the file data in sequence to acquire the absolute value of the corresponding second difference, and determine the second array data based on the absolute value of the second difference; Obtain a group of second arrays with the largest absolute value from the second array data, and obtain a first number of error data with the smallest absolute value from each second array corresponding to the file data, so as to determine a second target dimension array; Calculate the mean of each data in the second target dimension array to obtain a second mean array; Bubble sort is performed on the second mean array to obtain a second number of data with the smallest mean, and the second number of data is grouped according to the action names of free strength exercises, and whether the user's current action is an exercise ending action is determined based on the size relationship between the corresponding second grouping result and a preset threshold parameter.
6. The action recognition method according to claim 5, characterized in that: The determining whether the current action of the user is a motion ending action according to the size relationship between the corresponding second grouping result and a preset threshold parameter includes: If the number of the ending actions of the target movement in the free power movement in the second grouping result is greater than or equal to the preset threshold parameter, determining that the current action of the user is the movement ending action of the target movement of the free power movement; If the number of the ending actions of the target movement in the free power movement in the second grouping result is less than the preset threshold parameter, it is determined that the current action of the user is an invalid action.
7. The motion recognition method according to any one of claims 1 to 6, characterized in that: The determining the number of effective free force exercises of the user according to the first comparison result and the second comparison result includes: If the actions corresponding to the movement start action and the movement end action are judged to be the start or end of the same movement action, then the current free power exercise of the user is determined to be a valid action data, so as to determine the number of effective free power exercises of the user based on the judgment of the valid action data.
8. A motion recognition device, characterized in that: include: A database creation module, used to create a start action database and an end action database; A position information acquisition module is used to analyze the video data of the user completing the target movement frame by frame using a neural network algorithm model to obtain the position information of the sports equipment in each frame; a longitudinal displacement change determination module, used to determine the longitudinal displacement change of the sports equipment in the current frame and the sports equipment in the previous frame based on the position information; a first comparison module, configured to generate first dimensional data based on the target joint point information of the user in the picture corresponding to the current frame when the value corresponding to the longitudinal displacement change is not equal to zero, compare the first dimensional data with the file data in the start action database, and determine whether the current action of the user is a motion start action according to the corresponding first comparison result; A second comparison module is used to generate second dimensional data based on the target joint point information of the user in the picture corresponding to the current frame when the value corresponding to the longitudinal displacement change is equal to zero, compare the second dimensional data with the file data in the end action database, and determine whether the current action of the user is a movement end action according to the corresponding second comparison result; The exercise number determination module is used to determine the number of effective free-strength exercises of the user according to the first comparison result and the second comparison result.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the action recognition method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the action recognition method according to any one of claims 1 to 7.