Motion detection method, computer device, storage medium and computer program

Abstract

The present invention provides a motion action detection method, computer equipment, storage medium and computer program. The method includes parsing an action video to obtain an action image; inputting each action image into an action recognition model, and using the action recognition model to identify the action type of the object to be measured in the action image; using the Yolo-Pose human posture estimation algorithm to parse the action video to obtain the motion trajectory of the object to be measured in the action video; calculating the motion angle of the limb corresponding to the action type in the motion trajectory to obtain the limb action angle; when the limb action angle meets the preset action angle threshold corresponding to the action type, it is determined that an action is completed. Since the action type of the object to be measured in the video is first identified by the action recognition model, the limb to be measured that needs to be detected can be determined according to the action type. Focusing on the motion angle of the limb corresponding to the action type, there is no need to detect the motion angles of other limbs, and the action and the number of actions can be accurately detected.

Description

Technical Field

[0001] The present invention relates to the technical field of motion detection, and in particular to a motion action detection method, computer equipment, storage medium and computer program. Background Art

[0002] With the development of science and technology, motion recognition and detection through machine vision has gradually been widely used.

[0003] With the help of current machine vision-based motion recognition, in most cases, it can accurately identify the movement state of the human body. However, when it comes to detecting the standardization of user movements, its accuracy is low, and it is often unable to accurately identify the process of a user's single movement, resulting in inaccurate detection of the completion of the movement and inaccurate movement counting. Summary of the Invention

[0004] Based on this, it is necessary to provide a motion detection method, computer equipment, storage medium and computer program to address the above technical problems.

[0005] A motion detection method, comprising:

[0006] Get action videos;

[0007] parsing the action video to obtain a plurality of action images arranged in chronological order;

[0008] Inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type;

[0009] Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain the motion trajectory of the object to be measured in the action video;

[0010] Based on the motion type, calculating the motion angle of the limb corresponding to the motion type in the motion trajectory to obtain the limb motion angle;

[0011] Detecting whether the limb movement angle meets the preset movement angle threshold corresponding to the movement type;

[0012] When the limb movement angle meets the preset movement angle threshold corresponding to the movement type, it is determined that the movement corresponding to the movement type is completed once.

[0013] In one embodiment, the step of parsing the action video using the Yolo-Pose human posture estimation algorithm to obtain the motion trajectory of the object to be measured in the action video includes:

[0014] Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video;

[0015] detecting a motion range of each of the joint coordinates in a preset number of consecutive frames of the action image, and determining a limb relationship of each of the joint coordinates according to the motion range of each of the joint coordinates;

[0016] Based on the limb relationship, the limbs of the subject to be measured and the coordinates of the joint points corresponding to each limb are determined, and the motion trajectory of the limbs of the subject to be measured is determined according to the coordinates of the joint points corresponding to the limbs of the subject to be measured.

[0017] In one embodiment, the step of parsing the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video includes:

[0018] Analyzing the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of original joint coordinates of the object to be measured in the action video;

[0019] Determining a plurality of detection frames corresponding to the joints of the object to be measured, and extracting transverse and longitudinal parameters of the detection frames;

[0020] Calculating a horizontal relative ratio and a vertical relative ratio of each original joint coordinate to the starting coordinate of the detection frame using the horizontal parameters and the vertical parameters of each original joint coordinate and the corresponding detection frame;

[0021] According to the horizontal relative proportion and the vertical relative proportion, normalized feature processing is performed on the original joint coordinates to obtain the joint point coordinates.

[0022] In one embodiment, the step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type includes:

[0023] Inputting each of the action images into a pre-trained action recognition model in the time sequence, using the action recognition model to recognize the action of the subject to be tested in the action image, and obtaining the action type of the subject to be tested in each of the action images;

[0024] According to the action type of the object to be measured in each action image, the action images are grouped in the time sequence to obtain a plurality of groups of action series images;

[0025] Calculating motion parameters of the subject in each of the series of motion images belonging to the same group using a Yolo-Pose human pose estimation algorithm, and generating a parameter trend curve based on the time sequence based on the motion parameters in each of the series of motion images in the same group;

[0026] parsing the parameter trend curve to obtain peak points, trough points, and slopes corresponding to the peak points and trough points of the parameter trend curve, and determining an action start frame image and an action end frame image of the object to be measured in each of the action series images belonging to the same group based on the peak points, trough points, and slopes corresponding to the peak points and trough points of the parameter trend curve;

[0027] The step of detecting the motion range of each joint point coordinate in the action image of a preset number of consecutive frames, and determining the limb relationship of each joint point coordinate according to the motion range of each joint point coordinate comprises:

[0028] Detect the movement range of each joint point coordinate in the action image of a continuous preset number of frames, and determine the limb relationship of each joint point coordinate based on the movement range of each joint point coordinate, wherein the action image of a continuous preset number of frames is a continuous multi-frame action image from the action start frame image to the action end frame image in each action series image of the same group.

[0029] In one embodiment, the step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type includes:

[0030] Extracting M consecutive test images from the action video in the time sequence at intervals of N frames, where N and M are positive integers greater than 2, and M is less than N;

[0031] Using the Yolo-Pose human pose estimation algorithm to identify the joints of the subject to be measured in each of the test images, and marking the joints of each of the action images in the corresponding test images;

[0032] The test images marked with the joint points and other action images are input into a pre-trained action recognition model in the time sequence, and the action recognition model is used to recognize the action of the object to be tested in the action image to obtain the action type.

[0033] In one embodiment, the step of calculating the movement angle of a limb corresponding to the movement type in the movement trajectory based on the movement type to obtain the limb movement angle includes:

[0034] Based on the motion type, determining a plurality of reference limbs of the subject to be measured and a relative motion limb corresponding to each of the reference limbs;

[0035] According to the action type, each reference limb and the relative moving limb corresponding to each reference limb are tracked until the action image corresponding to the action type reaches a continuous preset number of frames, and a decomposed action trajectory is extracted from the motion trajectory;

[0036] The relative motion angle between the reference limb and the corresponding relative motion limb in the decomposed motion trajectory is calculated to obtain the limb motion angle.

[0037] In one embodiment, the method further comprises:

[0038] Get sample video;

[0039] Analyzing the sample video using a Yolo-Pose human pose estimation algorithm to obtain a preset number of sample joint coordinates of a sample object in the sample video;

[0040] Calculate the sample motion types and motion parameters of the sample object according to a preset number of sample joint coordinates;

[0041] Associating the sample action type and action parameters of the sample object with a plurality of sample images arranged in time series in the sample video;

[0042] The associated sample images and sample action types and action parameters of the sample objects are input into a support vector machine classification model, and the support vector machine classification model is trained to obtain the action recognition model.

[0043] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in any one of the above embodiments when executing the computer program.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in any one of the above embodiments.

[0045] A computer program, which implements the steps described in any one of the above embodiments when executed by a processor.

[0046] The above-mentioned motion detection method, computer device, storage medium, and computer program first use a motion recognition model to identify the motion type of the subject to be tested in the video, then use the Yolo-Pose human posture estimation algorithm to further analyze the motion to obtain the motion trajectory of the subject to be tested, and determine the limb to be detected based on the motion type. The angle of the limb is then detected using the motion trajectory, thereby accurately and efficiently detecting the motion and the number of motions of the subject to be tested. Because the motion recognition model is first used to first identify the motion type of the subject to be tested in the video, the limb to be tested can then be determined based on the motion type, thereby focusing on the motion angle of the limb corresponding to the motion type without having to detect the motion angles of other limbs, thereby improving detection efficiency and making the detection results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 1 is a flow chart of a motion detection method according to an embodiment;

[0048] Figure 2 is a diagram of the internal structure of a computer device in one embodiment;

[0049] Figure 3 A diagram of the process of detecting whether the angle of a limb movement is consistent in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] Example 1

[0052] In this embodiment, Figure 1 As shown, a motion detection method is provided, which includes:

[0053] Step 110: Obtain action video.

[0054] In this embodiment, the action video is a video containing a subject to be tested, wherein the subject to be tested is in motion in the video. The subject to be tested is a person in motion in the video, and is the person whose action is being detected. For example, the subject to be tested is a teenager in a youth sports assessment.

[0055] In order to improve the accuracy of recognition and detection, the action video in this embodiment is a video shot synchronously by two cameras. The two cameras shoot the object to be tested from different directions to obtain camera videos respectively. Therefore, in this embodiment, the action video includes two camera videos.

[0056] Step 120: parse the action video to obtain a plurality of action images arranged in chronological order.

[0057] In this embodiment, the action video is analyzed to obtain multiple frames of images of the action video, and the multiple frames of images are multiple action images arranged in chronological order.

[0058] Step 130 : Input each of the action images into a pre-trained action recognition model in the time sequence, and use the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type.

[0059] In this embodiment, the pre-trained action recognition model is a machine learning model that has been trained to recognize human actions in images. In one embodiment, the action recognition model is a support vector machine classification model.

[0060] In this embodiment, the action images are sequentially input into the action recognition model in chronological order. The action recognition model recognizes the action of the subject to be tested in the action images and obtains the type of action of the subject to be tested. In one embodiment, the action types include squats, pull-ups, push-ups, and flyes. It is worth mentioning that different action types have different human postures, limb movement trajectories, and limb rotation angles of the subject to be tested. The action recognition model identifies the action type of the subject to be tested, and then the movement angle of the limb corresponding to the action type can be determined based on the action type, thereby determining the key points that need to be detected for the subject to be tested. For example, in pull-ups, the torso and legs only move in the vertical direction, but the angles between the torso and legs and the angles between them do not change. In pull-ups, there is no need to detect the angles of the torso and legs. In the pull-up action, the arm movement amplitude is large. Therefore, for the action type of pull-ups, the corresponding limb is the arm, and it is necessary to detect the key points corresponding to the arm, such as the shoulder and elbow.

[0061] Step 140: parse the action video using a Yolo-Pose human posture estimation algorithm to obtain a motion trajectory of the object to be measured in the action video.

[0062] In this embodiment, the Yolo-Pose human pose estimation algorithm is used to analyze the action video to obtain the motion trajectory of each limb of the subject to be measured. Specifically, the Yolo-Pose human pose estimation algorithm is used to extract the joint points in the video, and the joint point coordinates are used to calculate the motion trajectory of the limbs of the subject to be measured.

[0063] Step 150 : Based on the motion type, calculate the motion angle of the limb corresponding to the motion type in the motion trajectory to obtain the limb motion angle.

[0064] It is worth mentioning that different motion types correspond to different limb movements, and the motion angles and amplitudes of the limbs corresponding to different motion types are different. Therefore, in order to improve detection efficiency and detection accuracy, in this embodiment, the correspondence between the motion type and the limb is first obtained. Based on the correspondence between the motion type and the limb, the limb corresponding to the motion type is determined as the limb to be tested, and the motion angle of the limb to be tested in the motion trajectory is calculated to obtain the limb motion angle. The motion angle of the limb in the motion trajectory is thus calculated. It should be understood that the object to be tested involves the movement of the trunk and multiple limbs during the motion process. If the motion angles of all limbs are detected, it will result in low efficiency and easily cause interference. Therefore, in this embodiment, only the motion angle of the limb to be tested corresponding to the motion type in the motion trajectory is calculated. For other limbs that do not correspond to the motion type, the subsequent steps of detection and calculation are not performed. On the one hand, the number of limb angles that need to be calculated is reduced. On the other hand, limbs that have little impact on the motion type are excluded, avoiding interference, thereby not only improving detection efficiency but also improving detection accuracy.

[0065] Step 160 , detecting whether the limb movement angle meets a preset movement angle threshold corresponding to the movement type.

[0066] In this embodiment, for each action type, if the action is completed in a standardized and qualified manner, the movement angle of the human limb must meet the standard, that is, the limb movement angle of the subject to be measured must meet the preset action angle threshold corresponding to the action type. In this embodiment, the limb movement angle meeting the preset action angle threshold corresponding to the action type means that the limb movement angle is greater than the preset action angle threshold corresponding to the action type. In some embodiments, the limb movement angle may also be less than the preset action angle threshold corresponding to the action type, depending on the requirements of the limb movement type.

[0067] For example, for push-ups, the angle between the first two hand key points and the last two key points is used as a reference standard. The angle between the first two hand key points and the last two key points is the limb movement angle. The angle threshold between the two lines (the preset movement angle threshold) is set to 90 degrees. In this way, a push-up is considered to be a qualified limb movement only if the limb movement angle is greater than 90 degrees. For other movement types, such as the bird fly movement, the corresponding limb angle must be greater than 75 degrees to be considered a qualified bird fly movement.

[0068] Step 170: When the limb movement angle meets the preset movement angle threshold corresponding to the movement type, it is determined that the movement corresponding to the movement type is completed once.

[0069] In this embodiment, when the limb movement angle meets the preset movement angle threshold corresponding to the movement type, the movement is determined to be completed once. In the video, if the object to be measured completes the movement multiple times, the number of movements is counted, thereby realizing automatic detection of the movement of the object to be measured.

[0070] Specific testing process, combined with Figure 3 First, the action type is identified by using the action recognition model, that is, action detection is performed. After detecting which of the four actions, squats, pull-ups, push-ups, and flyes, the limb movement angle of the object to be tested is detected according to the preset action angle threshold corresponding to the action type. For example, the preset action angle threshold of the flye action is 75°, and the counting method is max. The limb movement angle of the object to be tested needs to be greater than the preset action angle threshold of 75°. The limb movement angle is the angle between the six key points. When the limb movement angle is greater than the preset action angle threshold of 75°, it is determined that an action is completed, and the action number is increased by one.

[0071] For example, the preset action angle threshold for the squat action is 90°, and the counting method is min. The limb movement angle of the subject to be tested needs to be less than the preset action angle threshold of 90° to meet the standard. The limb movement angle is the angle between six key points (that is, the angle between the thigh and the calf). When the limb movement angle is less than the preset action angle threshold of 90°, it is determined that one action is completed, and the action number is increased by one.

[0072] According to the above process, until the end of the video, action detection is achieved and action counting is achieved.

[0073] In this embodiment, the action type of the object to be tested in the video is first identified using the action recognition model, and then the action is further analyzed using the Yolo-Pose human posture estimation algorithm to obtain the motion trajectory of the object to be tested, and the limb to be tested is determined based on the action type. The angle of the limb is then detected using the motion trajectory, thereby accurately and efficiently detecting the action and number of actions of the object to be tested. Since the action type of the object to be tested in the video is first identified using the action recognition model, the limb to be tested can then be determined based on the action type, thereby focusing on the movement angle of the limb corresponding to the action type without having to detect the movement angles of other limbs, thereby improving detection efficiency and making the detection results more accurate.

[0074] In addition, it is worth mentioning that the method of the present application is not only applicable to the detection of a user performing a single action, but also applicable to situations where the user performs different actions in an action video. For example, the user (the subject to be tested) irregularly performs squats, pull-ups, push-ups and flys. In this way, the present application can first use the action recognition model to separately identify the action types in different stages of the video, and determine the limbs that need to be detected for the action type, and detect the angle of the limb, thereby realizing the completion detection of multiple different actions appearing in the video.

[0075] In order to calculate the motion trajectory of the object to be measured, in one embodiment, the step of using the Yolo-Pose human posture estimation algorithm to analyze the action video and obtain the motion trajectory of the object to be measured in the action video includes:

[0076] Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video;

[0077] detecting a motion range of each of the joint coordinates in a preset number of consecutive frames of the action image, and determining a limb relationship of each of the joint coordinates according to the motion range of each of the joint coordinates;

[0078] Based on the limb relationship, the limbs of the subject to be measured and the coordinates of the joint points corresponding to each limb are determined, and the motion trajectory of the limbs of the subject to be measured is determined according to the coordinates of the joint points corresponding to the limbs of the subject to be measured.

[0079] In this embodiment, the preset number is 17. Specifically, the Yolo-Pose human posture estimation algorithm is used to parse the action video and extract 17 joint points of the object to be measured in the video. These 17 joint points are nose x1, eye x2, ear x2, shoulder x2, elbow x2, wrist x2, hip x2, knee x2, and ankle x2. The coordinates of these 17 joint points, i.e., joint point coordinates, are extracted, and the confidence of the joint point coordinates is also calculated.

[0080] It is worth mentioning that each limb can be regarded as a line connecting the joints of two positions. For example, the arm (upper arm) is a line connecting the shoulder and the elbow, and the arm (lower arm) is a line connecting the elbow and the wrist. During the movement, the coordinates of the joints change. The trajectory formed during the change is the movement trajectory of the corresponding limb. The movement trajectory is a set of coordinates of the joints corresponding to the limbs arranged in chronological order.

[0081] It should be understood that, according to the structure of the human body, the motion ranges of joints in different positions are different, and the motion ranges of adjacent joints are also different. The motion trends of adjacent joints are similar. For example, in a squat, the motion range of the arms is small, while the motion ranges of the thighs and calves are large. Therefore, it is possible to determine which limb these joints belong to based on the motion range of the joints and the motion ranges of adjacent joints. In this embodiment, the changes in the coordinates of the joints in a continuous preset number of frames are detected to obtain the motion ranges of the joints in these frame images. Based on the motion ranges of the coordinates of each joint, the limbs corresponding to each joint can be determined, and different joints belonging to the same limb can be divided according to the limb relationship. In this way, different joints and adjacent joints can be divided according to the limb relationship. For example, the limb relationship between the shoulder and elbow is the arm (upper arm), the limb relationship between the elbow and wrist is the arm (lower arm), and the limb relationship between the knee joint and ankle is the calf.

[0082] By determining the limb relationship of each joint point coordinate, we can determine which limb each joint point belongs to, and we can also determine the joint point coordinates corresponding to the limb. For example, the joint point coordinates corresponding to the lower leg are the knee joint coordinates and the ankle coordinates. In this way, the movement trajectory of the limb can be determined.

[0083] In this example, the coordinates of the subject's 17 joints are first extracted. The range of motion of these joints is then used to classify the joints. A limb relationship is established for the joints belonging to the same limb. This determines the corresponding joint coordinates for each limb, allowing the motion trajectory of that limb to be accurately calculated based on the changes in the joint coordinates over time. Compared to directly detecting the joint coordinates and calculating their motion trajectory, detecting the limb's motion trajectory can more effectively eliminate noise interference and improve robustness.

[0084] In one embodiment, the step of parsing the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video includes:

[0085] Analyzing the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of original joint coordinates of the object to be measured in the action video;

[0086] Determining a plurality of detection frames corresponding to the joints of the object to be measured, and extracting transverse and longitudinal parameters of the detection frames;

[0087] Calculating a horizontal relative ratio and a vertical relative ratio of each original joint coordinate to the starting coordinate of the detection frame using the horizontal parameters and the vertical parameters of each original joint coordinate and the corresponding detection frame;

[0088] According to the horizontal relative proportion and the vertical relative proportion, normalized feature processing is performed on the original joint coordinates to obtain the joint point coordinates.

[0089] It should be understood that due to differences in height and body shape between different subjects, there may be errors in motion detection for different subjects. To address the problem of reduced accuracy and robustness caused by insufficient adaptability of motion recognition models and human posture estimation algorithms to different body shapes, this embodiment maps the coordinates of the joint points of different human bodies to the same scale space, making the model more robust and reliable.

[0090] Specifically, in this embodiment, the Yolo-Pose human posture estimation algorithm is first used to analyze the action video to obtain the original joint coordinates of the object to be measured, and then the horizontal and vertical parameters of the detection frame are obtained, wherein the horizontal parameter is the width of the detection frame and the vertical parameter is the height of the detection frame. Specifically, please refer to Table 1 below. The normalized feature processing process is: calculate the x-axis coordinate of the joint point coordinate: the ratio of the difference between the horizontal coordinate of the original joint point coordinate and the horizontal coordinate of the starting coordinate of the detection frame to the width of the detection frame to obtain the horizontal coordinate of the normalized joint point coordinate; calculate the y-axis coordinate of the joint point coordinate: the ratio of the difference between the vertical coordinate of the original joint point coordinate and the vertical coordinate of the starting coordinate of the detection frame to the height of the detection frame to obtain the vertical coordinate of the normalized joint point coordinate.

[0091] Table 1 Feature normalization pseudocode

[0092]

[0093] In this way, the coordinates of different joint points of people with different heights and body shapes are mapped to the same coordinate system, which facilitates unified detection of people with different heights and body shapes, making the detection results accurate and reliable, and effectively improving the robustness.

[0094] In one embodiment, the step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type includes:

[0095] Inputting each of the action images into a pre-trained action recognition model in the time sequence, using the action recognition model to recognize the action of the subject to be tested in the action image, and obtaining the action type of the subject to be tested in each of the action images;

[0096] According to the action type of the object to be measured in each action image, the action images are grouped in the time sequence to obtain a plurality of groups of action series images;

[0097] Calculating motion parameters of the subject in each of the series of motion images belonging to the same group using a Yolo-Pose human pose estimation algorithm, and generating a parameter trend curve based on the time sequence based on the motion parameters in each of the series of motion images in the same group;

[0098] parsing the parameter trend curve to obtain peak points, trough points, and slopes corresponding to the peak points and trough points of the parameter trend curve, and determining an action start frame image and an action end frame image of the object to be measured in each of the action series images belonging to the same group based on the peak points, trough points, and slopes corresponding to the peak points and trough points of the parameter trend curve;

[0099] The step of detecting the motion range of each joint point coordinate in the action image of a preset number of consecutive frames, and determining the limb relationship of each joint point coordinate according to the motion range of each joint point coordinate comprises:

[0100] Detect the movement range of each joint point coordinate in the action image of a continuous preset number of frames, and determine the limb relationship of each joint point coordinate based on the movement range of each joint point coordinate, wherein the action image of a continuous preset number of frames is a continuous multi-frame action image from the action start frame image to the action end frame image in each action series image of the same group.

[0101] It should be understood that if the start and end frames of an action are not accurately identified, the identified action may end prematurely, causing the limb movement angle to fail to reach the preset action angle threshold and be determined to be incomplete. In the actual process, the current action of the subject to be tested is still continuing and has reached the preset action angle threshold. This situation will lead to inaccurate action detection times. Therefore, in order to avoid this situation and improve the accuracy of action angle detection, in this embodiment, the purpose of dividing the action image into multiple groups of action series images is to accurately identify the start and end frames of each action, so as to accurately identify the initiation and end of each action, thereby improving the calculation accuracy of the limb movement angle in the subsequent steps, and accurately detecting whether the subject to be tested has completed an action.

[0102] Specifically, an action video contains multiple action types, that is, the subject performs multiple actions in a video. Therefore, in this embodiment, the action images are classified using the action types identified by the action recognition model and arranged in chronological order in the action video. For example, the action types in the first 30 seconds of the first minute of the action video are: squat, squat, squat... and the action types in the last 30 seconds of the first minute are: push-up, push-up, push-up... In this way, the action images of each frame of the action video can be divided into multiple groups of action series images based on the action types in different time periods.

[0103] In this embodiment, after a plurality of groups of action series images are divided, action parameters are calculated for the action series images belonging to the same group. The action parameters include the joint coordinates of the object to be measured, the movement trajectory of the joint coordinates, the relative positions of adjacent joints, the angles of the lines connecting adjacent joints relative to the lines connecting reference joints, etc. The action parameters of the action series images of the same group are respectively mapped to the coordinate axes. For example, the horizontal axis of the coordinate axis is time, and the vertical axis is angle, or the horizontal axis of the coordinate axis is time, and the vertical axis is the relative distance of the joints, thereby generating a parameter trend curve of the action parameters on the coordinate axis. By analyzing the peak points, trough points and the slopes corresponding to these points of the parameter trend curve, the extreme values ​​of the joints during the movement process can be determined. For example, the maximum distance, the maximum angle value, the minimum distance, the minimum angle value, etc., so that the time corresponding to the trough point and the peak point can be accurately identified as the start time and the end time of the action, and then the action start frame and the action end frame of the reorganized action series image can be determined. In this way, when the limb relationship of the coordinates of each joint point of the object to be measured is subsequently calculated, the action angle can be identified for each group of images, and then the start and end of an action can be accurately identified. Based on the start and end of an action, the limb relationship can be accurately determined, and the movement of each limb relative to the reference limb in each action can be calculated, the end of the action can be accurately determined, and whether the object to be measured has completed the action can be accurately detected, thereby effectively improving the accuracy of action detection and counting.

[0104] In one embodiment, the step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type includes:

[0105] Extracting M consecutive test images from the action video in the time sequence at intervals of N frames, where N and M are positive integers greater than 2, and M is less than N;

[0106] Using the Yolo-Pose human pose estimation algorithm to identify the joints of the subject to be measured in each of the test images, and marking the joints of each of the action images in the corresponding test images;

[0107] The test images marked with the joint points and other action images are input into a pre-trained action recognition model in the time sequence, and the action recognition model is used to recognize the action of the object to be tested in the action image to obtain the action type.

[0108] In this embodiment, N frames are used as the interval for extracting test images. The duration of the N frames of action images is related to the action type, and the duration of the N frames of action images is close to or the same as the completion time of an action. In this way, each time the subject performs an action, M frames of test images for that action can be extracted. For example, M is 30. Each time the subject performs an action, the first 30 frames of action images are extracted as the test images.

[0109] In this embodiment, to improve the real-time recognition efficiency of the action recognition model, before the action image is input into the action recognition model, M consecutive test image frames are first extracted at intervals based on the approximate completion time of the action, thereby extracting M test image frames at the beginning of each action. Subsequently, the extracted test images are labeled with joint points using the Yolo-Pose human pose estimation algorithm, for example, by marking the name, coordinates, and limb of the joint point. In this way, labeled test images are formed. The labeled test images and other unlabeled action images are input into the action recognition model in the original chronological order for action recognition. Because the first M test images of each action are labeled with the relevant node points, the action recognition model can quickly and accurately identify the action based on the trajectory of the joint point marks, and predict the action of the next NM frames of action images. The action of the subsequent NM frames of action images is corrected based on the action recognized in the M frames, thereby effectively improving the accuracy of action recognition and the efficiency of action recognition. Compared with relying solely on the action recognition model to perform action recognition on all action images, using marked test images for auxiliary recognition can effectively improve recognition efficiency and accuracy.

[0110] It is worth mentioning that in this embodiment, joint point recognition and marking are not performed on all N frames of action images of each action, but only on the first M frames of images. The purpose of this is to avoid the low efficiency caused by the excessive computational load of the Yolo-Pose human pose estimation algorithm for joint point recognition and marking. It should be understood that, relative to the increased computational load and computational load, the accuracy of action recognition is not improved to the same extent as the increased computational load by using all N frames for joint point recognition and marking. That is, only using M consecutive frames of images with an interval of N frames can improve the accuracy of action recognition to 100% or close to 100%. Using the Yolo-Pose human pose estimation algorithm to recognize and mark joint points on all N frames of images does not significantly improve the accuracy, but significantly increases the computational load, resulting in a decrease in the efficiency of overall action type recognition. Therefore, in this embodiment, using the Yolo-Pose human pose estimation algorithm to mark the joint points of the extracted M frames of test images not only effectively improves the action recognition accuracy of the action recognition model, but also effectively improves the action recognition efficiency of the action recognition model.

[0111] In one embodiment, the step of inputting the test images marked with the joint points and the other action images into a pre-trained action recognition model in the chronological order, and using the action recognition model to recognize the action of the subject to be tested in the action image to obtain the action type includes:

[0112] The test images marked with the joint points and other action images are input into a pre-trained action recognition model in the time sequence, and the action recognition model is used to recognize the action of the object to be tested in the action image to obtain the action type.

[0113] According to the markings of each test image, the posture of the subject to be tested in the other action images is tracked, the key point areas of the subject to be tested in the other action images are marked, and sparse processing is performed based on the marking results of the key point areas, retaining the key pixels of the limbs of the subject to be tested corresponding to the markings to obtain a sparse image;

[0114] The test images and the sparse images marked with the joint points are input into a pre-trained motion recognition model, and the motion recognition model is used to identify the motion of the object to be tested in each test image and the motion of the object to be tested in each sparse image to obtain the motion type.

[0115] In this embodiment, sparse processing is used to remove redundant information in other action images and retain only key data points. Specifically, in this embodiment, the SIFT (Scale Invariant Feature Transform) algorithm is used, and the areas around the marked joints of the test image are used as key data points, and the background, clothing, etc. in the image are used as redundant information. According to the marking of the test image, posture tracking is performed on the action images of the NM frames located between the first M frames of the test image and the second M frames of the test image, and the key point areas of the action images of the NM frames located between the first M frames of the test image and the second M frames of the test image are marked. Subsequently, the action images of the NM frames located between the first M frames of the test image and the second M frames of the test image are sparsely processed after the key point areas are marked to obtain an image that retains the key pixel data of the key limbs, thereby obtaining a sparse image. It is worth mentioning that the marking of the key point area here is different from the joint point marking in the above embodiment. In the above embodiment, the joint point marking requires extremely high precision and the joint point position needs to be determined very clearly. In this embodiment, only the area around the joint point needs to be roughly marked. Because the larger the area retained later, the easier it is to identify the movement of the joint point. If the area is too small, the difficulty of recognition is increased, which is not conducive to improving efficiency. It should be understood that in this embodiment, only the area around the joint point needs to be identified. The purpose is to mark the approximate movement range of the joint point to facilitate the recognition of the action recognition model.

[0116] This process further reduces the amount of image data input to the action recognition model, improving the model's recognition efficiency. Furthermore, by reducing noise interference, the accuracy of action type detection can be further improved. It's worth noting that because the joints identified in the first M frames are already sufficiently accurate for action recognition, even if the subsequent NM frames lose detail, the resulting error is minimal.

[0117] In one embodiment, the step of calculating the movement angle of a limb corresponding to the movement type in the movement trajectory based on the movement type to obtain the limb movement angle includes:

[0118] Based on the motion type, determining a plurality of reference limbs of the subject to be measured and a relative motion limb corresponding to each of the reference limbs;

[0119] According to the action type, each reference limb and the relative moving limb corresponding to each reference limb are tracked until the action image corresponding to the action type reaches a continuous preset number of frames, and a decomposed action trajectory is extracted from the motion trajectory;

[0120] The relative motion angle between the reference limb and the corresponding relative motion limb in the decomposed motion trajectory is calculated to obtain the limb motion angle.

[0121] In this embodiment, the reference limb is a reference object for calculating the movement angle of the relatively moving limb, and the angle of the relatively moving limb is the angle between the reference limb and the reference. That is, the reference limb serves as the measurement basis for the angle of the limb, and the relatively moving limb is the limb that moves relative to the reference limb. In this embodiment, each action type corresponds to one or more reference limbs, and each relatively moving limb corresponds to a reference limb. In this embodiment, the reference limb corresponding to each action type and the relatively moving limb corresponding to the reference limb are pre-stored. Therefore, according to the determined action type, the reference limb and the corresponding relatively moving limb can be determined, and then the coordinates of the reference limb and the corresponding relatively moving limb in multiple frames of action images can be tracked. Specifically, the continuous preset number of action images are the multiple frames of action images from the action start frame image to the action end frame image in each action series image of the same group.

[0122] In this embodiment, according to the action type, each reference limb and the relative moving limb corresponding to each reference limb are tracked from the action start frame image to the action end frame image in the same group of action series images. When the action end frame image is tracked, the decomposed action trajectory is extracted from the motion trajectory, and the change in the limb action angle of the object to be tested in this section of the decomposed action trajectory is calculated. In this way, the limb action angle of a complete action of the object to be tested can be fully identified, thereby accurately detecting whether the object to be tested has completed an action completely, avoiding the problem of inaccurate action detection times caused by prematurely determining the end of the action due to inaccurate action completion detection. Through the above process, the inaccuracy of action detection times can be effectively avoided. In addition, since the number of frames corresponding to an action is determined, the interference of invalid actions and noise coordinates can also be effectively avoided, further improving the detection accuracy.

[0123] In order to train a stable, reliable, and accurate action recognition model, in one embodiment, the method further includes:

[0124] Get sample video;

[0125] Analyzing the sample video using a Yolo-Pose human pose estimation algorithm to obtain a preset number of sample joint coordinates of a sample object in the sample video;

[0126] Calculate the sample motion types and motion parameters of the sample object according to a preset number of sample joint coordinates;

[0127] Associating the sample action type and action parameters of the sample object with a plurality of sample images arranged in time series in the sample video;

[0128] The associated sample images and sample action types and action parameters of the sample objects are input into a support vector machine classification model, and the support vector machine classification model is trained to obtain the action recognition model.

[0129] In this embodiment, the sample object is a moving human body in the sample video, that is, a person whose action is detected. In this embodiment, the sample video includes videos from two camera positions.

[0130] In this embodiment, the Yolo-Pose human pose estimation algorithm is first used to extract the coordinates of 17 human joints from a sample video to obtain sample joint coordinates. The Yolo-Pose human pose estimation algorithm is then used to calculate the motion type of the sample subject based on the movement of a preset number of sample joint coordinates. The motion type of the sample subject is the sample motion type. Specifically, the limb relationship of each sample joint coordinate is determined based on the range of motion of the sample joint coordinates. The motion trajectory of the limb is then determined based on the sample joint coordinates determined by the limb relationship, thereby determining the motion type of the sample subject. Furthermore, motion parameters can be calculated, including the motion angle and amplitude of the sample subject's limbs. Subsequently, the sample image of each frame is associated with the corresponding sample action type and action parameters. This association process is to mark the sample image. For example, the sample object in a certain sample image is marked as a squat, and the bending angle of the legs in the squat is 90 degrees. For example, the sample object in another sample image is marked as a push-up, and the bending angle of the arms in the push-up is 120 degrees. Subsequently, the associated sample image and the sample action type and action parameters of the sample object are used as a sample data set, and the sample data set is input into the support vector machine classification model for training, thereby obtaining a motion recognition model.

[0131] Since the sample dataset not only marks the action type corresponding to the sample image, but also marks the action parameters of the limb corresponding to the action type, it can enrich the accuracy of the sample dataset of the support vector machine classification model, thereby effectively improving the robustness of the trained action recognition model.

[0132] In this embodiment, the sample data set is divided into a training set and a test set. The ratio usually adopted is that most of the samples in the data set (80%) are used for training, and the remaining part is used for testing (20%). The support vector machine model is trained using the training set to construct a support vector machine classifier. After training is completed, the trained support vector machine model is tested using the test set. During the testing process, the sample features of the test set are input into the trained model, and classification predictions are made through the decision function of the model. The results are compared with the true labels of the test set to evaluate the performance of the model.

[0133] The current results show:

[0134] - Of the samples in category 1 (squats), all 277 were correctly classified as squats.

[0135] - Of the samples in category 2 (pull-ups), all 257 were correctly classified as pull-ups.

[0136] - Of the samples in category 3 (push-ups), all 278 were correctly classified as push-ups.

[0137] - Of the samples of category 4 (flying birds), all 148 were correctly classified as flying birds.

[0138] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0139] Example 2

[0140] In this embodiment, a motion detection device is provided, comprising:

[0141] An action video acquisition module, used to acquire action videos;

[0142] An action image parsing module is used to parse the action video to obtain a plurality of action images arranged in chronological order;

[0143] an action type recognition module, configured to input each of the action images into a pre-trained action recognition model in the time sequence, and use the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type;

[0144] A motion trajectory acquisition module is used to analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain the motion trajectory of the object to be measured in the action video;

[0145] a limb movement angle obtaining module, configured to calculate, based on the movement type, the movement angle of the limb corresponding to the movement type in the movement trajectory, to obtain the limb movement angle;

[0146] A comparison module, configured to detect whether the limb movement angle meets a preset movement angle threshold corresponding to the movement type;

[0147] The determination module is used to determine that the action corresponding to the action type is completed when the limb action angle meets the preset action angle threshold corresponding to the action type.

[0148] For the specific definition of the motion detection device, please refer to the definition of the motion detection method above, which will not be repeated here. The various units in the above-mentioned motion detection device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned units.

[0149] Example 3

[0150] In this embodiment, a computer device is provided. The computer device may be a portable wearable device. In this embodiment, the computer device is a smart wristband. The internal structure diagram thereof may be as follows: Figure 2As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores a computer program, and the non-volatile storage medium is used to store the user's action video and action image. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other computer devices, such as connecting to a server. The network interface is a wireless network interface that can access a mobile communication network and connect to a server. When the computer program is executed by the processor, a motion detection method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. In some embodiments, the input device includes a touch sensor, and the input device includes a photoelectric sensor, a skin electrical sensor, an inertial measurement unit (IMU), a temperature sensor and a sound sensor.

[0151] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0152] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps described in any of the above embodiments when executing the computer program.

[0153] Example 4

[0154] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps described in any of the above embodiments are implemented.

[0155] Example 5

[0156] In this embodiment, a computer program is provided. When the computer program is executed by a processor, the steps described in any of the above embodiments are implemented.

[0157] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0158] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A motion detection method, characterized in that: include: Get action videos; parsing the action video to obtain a plurality of action images arranged in chronological order; Inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type; Analyzing the action video using the Yolo-Pose human posture estimation algorithm to obtain the motion trajectory of each limb of the subject to be measured in the action video; Based on the motion type, calculating the motion angle of the limb corresponding to the motion type in the motion trajectory to obtain the limb motion angle; Detecting whether the limb movement angle meets the preset movement angle threshold corresponding to the movement type; When the limb movement angle meets the preset movement angle threshold corresponding to the movement type, determining that the movement corresponding to the movement type is completed once; The step of calculating the movement angle of the limb corresponding to the movement type in the movement trajectory based on the movement type to obtain the limb movement angle includes: Based on the motion type, determining a plurality of reference limbs of the subject to be measured and a relative motion limb corresponding to each of the reference limbs; According to the action type, each reference limb and the relative moving limb corresponding to each reference limb are tracked until the action image corresponding to the action type reaches a continuous preset number of frames, and a decomposed action trajectory is extracted from the motion trajectory; The relative motion angle between the reference limb and the corresponding relative motion limb in the decomposed motion trajectory is calculated to obtain the limb motion angle.

2. The method according to claim 1, characterized in that The step of using the Yolo-Pose human posture estimation algorithm to analyze the action video and obtain the motion trajectory of the object to be measured in the action video includes: Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video; detecting a motion range of each of the joint coordinates in a preset number of consecutive frames of the action image, and determining a limb relationship of each of the joint coordinates according to the motion range of each of the joint coordinates; Based on the limb relationship, the limbs of the subject to be measured and the coordinates of the joint points corresponding to each limb are determined, and the motion trajectory of the limbs of the subject to be measured is determined according to the coordinates of the joint points corresponding to the limbs of the subject to be measured.

3. The method according to claim 2, characterized in that The step of using the Yolo-Pose human posture estimation algorithm to analyze the action video and obtain the coordinates of a preset number of joint points of the object to be measured in the action video includes: Analyzing the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of original joint coordinates of the object to be measured in the action video; Determining a plurality of detection frames corresponding to the joints of the object to be measured, and extracting transverse and longitudinal parameters of the detection frames; Calculating a horizontal relative ratio and a vertical relative ratio of each original joint coordinate to the starting coordinate of the detection frame using the horizontal parameters and the vertical parameters of each original joint coordinate and the corresponding detection frame; According to the horizontal relative proportion and the vertical relative proportion, normalized feature processing is performed on the original joint coordinates to obtain the joint point coordinates.

4. The method according to claim 2, characterized in that The step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type includes: Inputting each of the action images into a pre-trained action recognition model in the time sequence, using the action recognition model to recognize the action of the subject to be tested in the action image, and obtaining the action type of the subject to be tested in each of the action images; According to the action type of the object to be measured in each action image, the action images are grouped in the time sequence to obtain a plurality of groups of action series images; Calculating motion parameters of the subject in each of the series of motion images belonging to the same group using a Yolo-Pose human pose estimation algorithm, and generating a parameter trend curve based on the time sequence based on the motion parameters in each of the series of motion images in the same group; parsing the parameter trend curve to obtain peak points, trough points, and slopes corresponding to the peak points and trough points of the parameter trend curve, and determining an action start frame image and an action end frame image of the object to be measured in each of the action series images belonging to the same group based on the peak points, trough points, and slopes corresponding to the peak points and trough points of the parameter trend curve; The step of detecting the motion range of each joint point coordinate in the action image of a preset number of consecutive frames, and determining the limb relationship of each joint point coordinate according to the motion range of each joint point coordinate comprises: Detect the movement range of each joint point coordinate in the action image of a continuous preset number of frames, and determine the limb relationship of each joint point coordinate based on the movement range of each joint point coordinate, wherein the action image of a continuous preset number of frames is a continuous multi-frame action image from the action start frame image to the action end frame image in each action series image of the same group.

5. The method according to claim 1, wherein The step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type includes: Extracting M consecutive test images from the action video in the time sequence at intervals of N frames, where N and M are positive integers greater than 2, and M is less than N; Using the Yolo-Pose human pose estimation algorithm to identify the joints of the subject to be measured in each of the test images, and marking the joints of each of the action images in the corresponding test images; The test images marked with the joint points and other action images are input into a pre-trained action recognition model in the time sequence, and the action recognition model is used to recognize the action of the object to be tested in the action image to obtain the action type.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Get sample video; Analyzing the sample video using a Yolo-Pose human pose estimation algorithm to obtain a preset number of sample joint coordinates of a sample object in the sample video; Calculate the sample motion types and motion parameters of the sample object according to a preset number of sample joint coordinates; Associating the sample action type and action parameters of the sample object with a plurality of sample images arranged in time series in the sample video; The associated sample images and sample action types and action parameters of the sample objects are input into a support vector machine classification model, and the support vector machine classification model is trained to obtain the action recognition model.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images