Motion action detection method, computer equipment, storage medium and computer program

Abstract

The invention provides a motion action detection method, computer equipment, a storage medium and a computer program, and the method comprises the steps: analyzing an action video, and obtaining an action image; inputting each action image into an action recognition model, and recognizing the action type of the to-be-detected object in the action image by using the action recognition model; analyzing the action video by using a Yolo-Pose human body posture estimation algorithm to obtain a motion track of the to-be-detected object in the action video; calculating a limb movement angle corresponding to the movement type in the movement track to obtain a limb movement angle; when the limb action angle conforms to a preset action angle threshold value corresponding to the action type, it is determined that one action is completed. The action type of the to-be-detected object in the video is firstly identified by using the action identification model, so that the to-be-detected limb of the to-be-detected object can be determined according to the action type, the motion angle of the limb corresponding to the action type is focused, the motion angles of other limbs do not need to be detected, and the detection efficiency is improved. And the action and the action times can be accurately detected.

Description

Technical Field

[0001] The present invention relates to the technical field of motion detection, and in particular to a motion action detection method, a computer device, a storage medium and a computer program. Background Art

[0002] With the development of science and technology, motion recognition and detection through machine vision has gradually been widely used.

[0003] With the help of current machine vision-based motion recognition, in most cases, the motion state of the human body can be accurately identified. However, when it comes to detecting the standardization of user movements, its accuracy is low, and it is often unable to accurately identify the process of a user's single movement, resulting in inaccurate detection of the completion of the movement and inaccurate movement counting. Summary of the invention

[0004] Based on this, it is necessary to provide a motion detection method, computer equipment, storage medium and computer program to address the above technical issues.

[0005] A motion detection method, comprising:

[0006] Get action video;

[0007] parsing the action video to obtain a plurality of action images arranged in chronological order;

[0008] Inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type;

[0009] The action video is analyzed using the Yolo-Pose human posture estimation algorithm to obtain a motion trajectory of the object to be measured in the action video;

[0010] Based on the action type, calculating the movement angle of the limb corresponding to the action type in the motion trajectory to obtain the limb movement angle;

[0011] Detecting whether the limb movement angle meets a preset movement angle threshold corresponding to the movement type;

[0012] When the limb movement angle meets the preset movement angle threshold corresponding to the movement type, it is determined that the movement corresponding to the movement type is completed once.

[0013] In one embodiment, the step of parsing the action video using the Yolo-Pose human posture estimation algorithm to obtain the motion trajectory of the object to be measured in the action video includes:

[0014] Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video;

[0015] Detecting the motion range of each of the joint point coordinates in the action images of a preset number of consecutive frames, and determining the limb relationship of each of the joint point coordinates according to the motion range of each of the joint point coordinates;

[0016] Based on the limb relationship, the limbs of the object to be measured and the coordinates of the joint points corresponding to each limb are determined, and the movement trajectory of the limbs of the object to be measured is determined according to the coordinates of the joint points corresponding to the limbs of the object to be measured.

[0017] In one embodiment, the step of parsing the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video includes:

[0018] Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of original joint coordinates of the object to be measured in the action video;

[0019] Determine a plurality of detection frames corresponding to the joints of the object to be detected, and extract the horizontal parameters and the vertical parameters of the detection frames;

[0020] Calculating the horizontal relative ratio and the vertical relative ratio of each of the original joint coordinates to the starting coordinates of the detection frame by using the horizontal parameters and the vertical parameters of each of the original joint coordinates and the corresponding detection frame;

[0021] According to the horizontal relative proportion and the vertical relative proportion, the original joint coordinates are subjected to normalized feature processing to obtain the joint point coordinates.

[0022] In one embodiment, the step of inputting each of the action images into a pre-trained action recognition model in the time sequence, using the action recognition model to recognize the action of the object to be tested in the action image, and obtaining the action type includes:

[0023] Inputting each of the action images into a pre-trained action recognition model in the time sequence, using the action recognition model to recognize the action of the object to be tested in the action image, and obtaining the action type of the object to be tested in each of the action images;

[0024] According to the action type of the object to be measured in each action image, grouping each action image in the time sequence to obtain multiple groups of action series images;

[0025] Calculating the motion parameters of the object to be measured in each of the action series images belonging to the same group by using the Yolo-Pose human posture estimation algorithm, and generating a parameter trend curve based on the time sequence based on the motion parameters in each of the action series images belonging to the same group;

[0026] Analyze the parameter trend curve to obtain the peak points, trough points and slopes corresponding to the peak points and trough points of the parameter trend curve, and determine the action start frame image and the action end frame image of the object to be measured in each of the action series images belonging to the same group based on the peak points, trough points and slopes corresponding to the peak points and trough points of the parameter trend curve;

[0027] The step of detecting the motion range of each of the joint point coordinates in the action images of a preset number of consecutive frames, and determining the limb relationship of each of the joint point coordinates according to the motion range of each of the joint point coordinates comprises:

[0028] Detect the movement range of each joint point coordinate in the action image of a continuous preset frame number, and determine the limb relationship of each joint point coordinate based on the movement range of each joint point coordinate, wherein the action image of a continuous preset frame number is a continuous plurality of frames of the action images from the action start frame image to the action end frame image in each action series image of the same group.

[0029] In one embodiment, the step of inputting each of the action images into a pre-trained action recognition model in the time sequence, using the action recognition model to recognize the action of the object to be tested in the action image, and obtaining the action type includes:

[0030] Extracting M consecutive test images from the action video in the time sequence at intervals of N frames, wherein N and M are positive integers greater than 2, and M is less than N;

[0031] Using the Yolo-Pose human posture estimation algorithm to identify the joint points of the object to be tested in each of the test images, and marking the joint points of each of the action images in the corresponding test image;

[0032] The test images marked with the joint points and other action images are input into a pre-trained action recognition model in the time sequence, and the action recognition model is used to recognize the action of the object to be tested in the action image to obtain the action type.

[0033] In one embodiment, the step of calculating the movement angle of a limb corresponding to the movement type in the movement trajectory based on the movement type to obtain the limb movement angle comprises:

[0034] Based on the action type, determining a plurality of reference limbs of the subject to be tested and a relative moving limb corresponding to each of the reference limbs;

[0035] According to the action type, each of the reference limbs and the relative moving limbs respectively corresponding to each of the reference limbs are tracked until the action image corresponding to the action type reaches a continuous preset number of frames, and a decomposed action trajectory is extracted from the motion trajectory;

[0036] The relative motion angle between the reference limb and the corresponding relative motion limb in the decomposed motion trajectory is calculated to obtain the limb motion angle.

[0037] In one embodiment, the method further comprises:

[0038] Get sample video;

[0039] Analyze the sample video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of sample joint coordinates of the sample object in the sample video;

[0040] Calculate the sample motion types and motion parameters of the sample object according to a preset number of sample joint coordinates;

[0041] Associating the sample action type and action parameters of the sample object with a plurality of sample images arranged in time series in the sample video;

[0042] The sample image and the sample action type and action parameters of the sample object after association are input into a support vector machine classification model, and the support vector machine classification model is trained to obtain the action recognition model.

[0043] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in any one of the above embodiments when executing the computer program.

[0044] A computer-readable storage medium stores a computer program, which implements the steps described in any of the above embodiments when executed by a processor.

[0045] A computer program, when executed by a processor, implements the steps described in any one of the above embodiments.

[0046] The above-mentioned motion detection method, computer device, storage medium and computer program use the motion recognition model to first identify the motion type of the object to be tested in the video, and then use the Yolo-Pose human posture estimation algorithm to further analyze the motion to obtain the motion trajectory of the object to be tested, and determine the limbs to be detected according to the motion type, and then use the motion trajectory to detect the angle of the limbs, so as to accurately and efficiently detect the motion and the number of motions of the object to be tested. Since the motion recognition model is first used to first identify the motion type of the object to be tested in the video, the limbs to be detected by the object to be tested can be determined according to the motion type, so as to focus on the motion angle of the limb corresponding to the motion type, without detecting the motion angles of other limbs, thereby improving the detection efficiency and making the detection result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic diagram of a flow chart of a motion detection method in an embodiment;

[0048] Figure 2 is an internal structure diagram of a computer device in one embodiment;

[0049] Figure 3 A diagram of the detection process of whether the body movement angle is consistent in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] Embodiment 1

[0052] In this embodiment, Figure 1 As shown, a motion detection method is provided, which includes:

[0053] Step 110, obtaining action video.

[0054] In this embodiment, the action video is a video containing the object to be tested, in which the object to be tested is in motion. The object to be tested is a moving human body in the video, and is the person whose action is being detected. For example, the object to be tested is a teenager in a youth sports assessment.

[0055] In order to improve the accuracy of recognition and detection, the action video in this embodiment is a video shot synchronously by two cameras. The two cameras shoot the object to be detected from different directions to obtain camera videos respectively. Therefore, in this embodiment, the action video includes two camera videos.

[0056] Step 120, parsing the action video to obtain a plurality of action images arranged in chronological order.

[0057] In this embodiment, the action video is analyzed to obtain multiple frames of images of the action video, and the multiple frames of images are multiple action images arranged in time sequence.

[0058] Step 130 , inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type.

[0059] In this embodiment, the pre-trained action recognition model is a machine learning model that has been trained to recognize the actions of a human body in an image. In one embodiment, the action recognition model is a support vector machine classification model.

[0060] In this embodiment, the action images are sequentially input into the action recognition model in chronological order, and the action recognition model recognizes the action of the object to be tested in the action images to obtain the type of action of the object to be tested. In one embodiment, the action type includes squats, pull-ups, push-ups and flyes. It is worth mentioning that different action types have different human postures, limb movement trajectories, and limb rotation angles of the object to be tested. The action type of the object to be tested is identified by the action recognition model, and then the movement angle of the limb corresponding to the sitting type can be determined according to the action type, so as to determine the key points that need to be detected by the object to be tested. For example, in pull-ups, the trunk and legs only move in the vertical direction, but the angles of the trunk and legs and the angles between them do not change. In pull-ups, there is no need to detect the angles of the trunk and legs. In the action of pull-ups, the movement amplitude of the arms is large. Therefore, for the action type of pull-ups, the corresponding limbs are arms, and it is necessary to detect the key points corresponding to the arms, such as shoulders and elbows.

[0061] Step 140: parse the action video using the Yolo-Pose human posture estimation algorithm to obtain a motion trajectory of the object to be measured in the action video.

[0062] In this embodiment, the Yolo-Pose human posture estimation algorithm is used to analyze the action video, so as to obtain the motion trajectory of each limb of the object to be measured. Specifically, the Yolo-Pose human posture estimation algorithm is used to extract the joint points in the video, and the joint point coordinates are used to calculate the motion trajectory of the limbs of the object to be measured.

[0063] Step 150, based on the action type, calculate the movement angle of the limb corresponding to the action type in the motion trajectory to obtain the limb movement angle.

[0064] It is worth mentioning that different action types correspond to different limb movements, and different action types correspond to different limb movement angles and amplitudes. Therefore, in order to improve detection efficiency and detection accuracy, in this embodiment, the corresponding relationship between the movement type and the limb is first obtained, and according to the corresponding relationship between the movement type and the limb, the limb corresponding to the movement type is determined as the limb to be tested, and the movement angle of the limb to be tested in the motion trajectory is calculated to obtain the limb movement angle. Thus, the movement angle of the limb in the motion trajectory is calculated. It should be understood that the object to be tested involves the movement of the trunk and multiple limbs during the movement process. If the movement angles of all limbs are detected, it will lead to low efficiency and easy interference. Therefore, in this embodiment, only the movement angle of the limb to be tested corresponding to the action type in the motion trajectory is calculated, and for other limbs that do not correspond to the action type, the detection and calculation of the subsequent steps are not performed. On the one hand, the number of limb angles that need to be calculated is reduced, and on the other hand, the limbs that have little effect on the movement type are excluded to avoid interference, thereby not only improving the detection efficiency, but also improving the detection accuracy.

[0065] Step 160, detecting whether the body movement angle meets a preset movement angle threshold corresponding to the movement type.

[0066] In this embodiment, for each action type, if the action is completed in a standardized and qualified manner, the movement angle of the human body's limbs must meet the standard, that is, the limb movement angle of the object to be measured needs to meet the preset action angle threshold corresponding to the action type. In this embodiment, the limb movement angle meets the preset action angle threshold corresponding to the action type, which means that the limb movement angle is greater than the preset action angle threshold corresponding to the action type. In some embodiments, the limb movement angle may also be less than the preset action angle threshold corresponding to the action type, depending on the requirements of the limb movement type.

[0067] For example, for push-ups, the angle between the first two key points of the hand and the last two key points is used as a reference standard. The angle between the first two key points of the hand and the last two key points is the limb movement angle. The angle threshold of the two lines (preset action angle threshold) is set to 90 degrees. In this way, a limb movement angle greater than 90 degrees is considered a qualified push-up. In other types of movements, such as the flying bird movement, the corresponding limb angle needs to be greater than 75 degrees to be considered a qualified flying bird movement.

[0068] Step 170: When the body movement angle meets the preset movement angle threshold corresponding to the movement type, it is determined that the movement corresponding to the movement type is completed once.

[0069] In this embodiment, when the limb movement angle meets the preset movement angle threshold corresponding to the movement type, the movement is determined to be completed once. In the video, if the object to be measured completes the movement multiple times, the number of movements is counted, thereby realizing automatic detection of the movement of the object to be measured.

[0070] Specific testing process, combined with Figure 3 First, the action type is identified by using the action recognition model, that is, action detection is performed. After detecting which of the four actions, squats, pull-ups, push-ups and flyes, the limb action angle of the object to be tested is detected according to the preset action angle threshold corresponding to the action type. For example, the preset action angle threshold of the fly action is 75°, and the counting method is max. The limb action angle of the object to be tested needs to be greater than the preset action angle threshold of 75°. The limb action angle is the angle between the six key points. When the limb action angle is greater than the preset action angle threshold of 75°, it is determined that an action is completed, and the action number is increased by one.

[0071] For example, the preset action angle threshold of the squat action is 90°, and the counting method is min. The limb movement angle of the subject to be tested needs to be less than the preset action angle threshold of 90° to meet the standard. The limb movement angle is the angle between six key points (i.e., the angle between the thigh and the calf). When the limb movement angle is less than the preset action angle threshold of 90°, it is determined that an action is completed, and the action number is increased by one.

[0072] According to the above process, until the video ends, action detection is achieved and action counting is realized.

[0073] In this embodiment, the action type of the object to be tested in the video is first identified using the action recognition model, and then the action is further analyzed using the Yolo-Pose human posture estimation algorithm to obtain the motion trajectory of the object to be tested, and the limbs to be detected are determined according to the action type, and then the angles of the limbs are detected using the motion trajectory, so as to accurately and efficiently detect the action and the number of actions of the object to be tested. Since the action type of the object to be tested in the video is first identified using the action recognition model, the limbs to be detected by the object to be tested can be determined according to the action type, so as to focus on the movement angle of the limb corresponding to the action type, without detecting the movement angles of other limbs, thereby improving the detection efficiency and making the detection results more accurate.

[0074] In addition, it is worth mentioning that the method of the present application is not only applicable to the detection of a single action performed by a user, but also to the situation where the user performs different actions in an action video. For example, the user (the subject to be tested) irregularly performs squats, pull-ups, push-ups and flys. In this way, the present application can first use the action recognition model to separately identify the action types in different stages of the video, and determine the limbs that need to be detected for the action type, and detect the angle of the limb, thereby realizing the completion detection of a variety of different actions appearing in the video.

[0075] In order to calculate the motion trajectory of the object to be measured, in one embodiment, the step of parsing the action video using the Yolo-Pose human posture estimation algorithm to obtain the motion trajectory of the object to be measured in the action video includes:

[0076] Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video;

[0077] Detecting the motion range of each of the joint point coordinates in the action images of a preset number of consecutive frames, and determining the limb relationship of each of the joint point coordinates according to the motion range of each of the joint point coordinates;

[0078] Based on the limb relationship, the limbs of the object to be measured and the coordinates of the joint points corresponding to each limb are determined, and the movement trajectory of the limbs of the object to be measured is determined according to the coordinates of the joint points corresponding to the limbs of the object to be measured.

[0079] In this embodiment, the preset number is 17. Specifically, the Yolo-Pose human posture estimation algorithm is used to parse the action video, and 17 joint points of the object to be measured in the video are extracted, and the 17 joint points are nose x1, eye x2, ear x2, shoulder x2, elbow x2, wrist x2, hip x2, knee x2, ankle x2, and the coordinates of the 17 joint points are extracted, that is, the joint point coordinates, and the confidence of the joint point coordinates is also calculated.

[0080] It is worth mentioning that each limb can be regarded as a line connecting the joints of two positions. For example, the arm (upper arm) is a line connecting the shoulder and the elbow, and the arm (lower arm) is a line connecting the elbow and the wrist. During the movement, the coordinates of the joints change. The trajectory formed during the change is the motion trajectory of the corresponding limb, which is a set of coordinates of the joints corresponding to the limbs arranged in chronological order.

[0081] It should be understood that, according to the structure of the human body, the joints at different positions have different ranges of motion, and the ranges of motion of the joints adjacent to them are also different. The motion trends of the adjacent joints are similar. For example, in the squatting action, the range of motion of the arm is small, while the range of motion of the thigh and calf is large. Therefore, it is possible to determine which limb these joints belong to based on the range of motion of the joints and the range of motion of the adjacent joints. In this embodiment, the changes in the coordinates of the joints in the images of the continuous preset number of frames are detected to obtain the range of motion of the joints in these frame images. According to the range of motion of the coordinates of each joint, the limbs corresponding to each joint can be determined, and the different joints belonging to the same limb can be divided according to the limb relationship. In this way, different joints and adjacent joints can be divided according to the limb relationship. For example, the limb relationship between the shoulder and the elbow is the arm (upper arm of the hand), the limb relationship between the elbow and the wrist is the arm (lower arm of the hand), and the limb relationship between the knee joint and the ankle is the calf.

[0082] By determining the limb relationship of each joint point coordinate, it is determined which limb each joint point belongs to, and the joint point coordinates corresponding to the limb can also be determined. For example, the joint point coordinates corresponding to the lower leg are the knee joint coordinates and the ankle coordinates. In this way, the movement trajectory of the limb can be determined.

[0083] In this embodiment, the coordinates of the 17 joints of the object to be tested are first extracted, and the motion range of these joints is used to divide different joints, and the joints belonging to the same limb are established to establish limb relationships, so as to determine the coordinates of the joints corresponding to the limb, so that the motion trajectory of the limb can be accurately calculated according to the change of the joint coordinates over time. Compared with directly detecting the coordinates of the joints and calculating the motion trajectory of the joints, detecting the motion trajectory of the limb can more effectively eliminate noise interference and improve robustness.

[0084] In one embodiment, the step of parsing the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video includes:

[0085] Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of original joint coordinates of the object to be measured in the action video;

[0086] Determine a plurality of detection frames corresponding to the joints of the object to be detected, and extract the horizontal parameters and the vertical parameters of the detection frames;

[0087] Calculating the horizontal relative ratio and the vertical relative ratio of each of the original joint coordinates to the starting coordinates of the detection frame by using the horizontal parameters and the vertical parameters of each of the original joint coordinates and the corresponding detection frame;

[0088] According to the horizontal relative proportion and the vertical relative proportion, the original joint coordinates are subjected to normalized feature processing to obtain the joint point coordinates.

[0089] It should be understood that due to the differences in height and body shape of different objects to be measured, there may be errors in the action detection of different objects to be measured. To this end, in order to solve the problem of reduced accuracy and robustness caused by the lack of adaptability of action recognition models and human posture estimation algorithms to different body shapes. In this embodiment, the model is made more robust and reliable by mapping the coordinates of the joint points of different human bodies to the same scale space.

[0090] Specifically, in this embodiment, the Yolo-Pose human posture estimation algorithm is first used to parse the action video to obtain the original joint coordinates of the object to be measured, and then the horizontal parameter and the vertical parameter of the detection frame are obtained, wherein the horizontal parameter is the width of the detection frame and the vertical parameter is the height of the detection frame. Specifically, please refer to Table 1 below, the normalized feature processing process is: calculate the x-axis coordinate of the joint point coordinate: the ratio of the difference between the horizontal coordinate of the original joint point coordinate and the horizontal coordinate of the starting coordinate of the detection frame to the width of the detection frame, and obtain the horizontal coordinate of the normalized joint point coordinate; calculate the y-axis coordinate of the joint point coordinate: the ratio of the difference between the vertical coordinate of the original joint point coordinate and the vertical coordinate of the starting coordinate of the detection frame to the height of the detection frame, and obtain the vertical coordinate of the normalized joint point coordinate.

[0091] Table 1 Feature normalization pseudo code

[0092]

[0093] In this way, the coordinates of different joint points of people with different heights and body shapes are mapped to the same coordinate system, which facilitates unified detection of people with different heights and body shapes, making the detection results accurate and reliable, and effectively improving the robustness.

[0094] In one embodiment, the step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type includes:

[0095] Inputting each of the action images into a pre-trained action recognition model in the time sequence, using the action recognition model to recognize the action of the object to be tested in the action image, and obtaining the action type of the object to be tested in each of the action images;

[0096] According to the action type of the object to be measured in each action image, grouping each action image in the time sequence to obtain multiple groups of action series images;

[0097] Calculating the motion parameters of the object to be measured in each of the action series images belonging to the same group by using the Yolo-Pose human posture estimation algorithm, and generating a parameter trend curve based on the time sequence based on the motion parameters in each of the action series images belonging to the same group;

[0098] Analyze the parameter trend curve to obtain the peak points, trough points and slopes corresponding to the peak points and trough points of the parameter trend curve, and determine the action start frame image and the action end frame image of the object to be measured in each of the action series images belonging to the same group based on the peak points, trough points and slopes corresponding to the peak points and trough points of the parameter trend curve;

[0099] The step of detecting the motion range of each of the joint point coordinates in the action images of a preset number of consecutive frames, and determining the limb relationship of each of the joint point coordinates according to the motion range of each of the joint point coordinates comprises:

[0100] Detect the movement range of each joint point coordinate in the action image of a continuous preset frame number, and determine the limb relationship of each joint point coordinate based on the movement range of each joint point coordinate, wherein the action image of a continuous preset frame number is a continuous plurality of frames of the action images from the action start frame image to the action end frame image in each action series image of the same group.

[0101] It should be understood that if the start frame and the end frame of the action are not accurately identified, the identified action may end prematurely, causing the limb movement angle to fail to reach the preset action angle threshold and be determined as unfinished. In the actual process, the current action of the object to be tested is still continuing and reaches the preset action angle threshold. This situation will lead to inaccurate action detection times. Therefore, in order to avoid this situation and improve the accuracy of action angle detection, in this embodiment, the purpose of dividing the action image into multiple groups of action series images is to accurately identify the start and end frames of each action, so that the initiation and end of each action can be accurately identified, thereby improving the calculation accuracy of the limb movement angle in the subsequent steps, and accurately detecting whether the object to be tested has completed an action.

[0102] Specifically, a plurality of action types are included in an action video, that is, the object to be tested performs a plurality of action movements in a video. Therefore, in this embodiment, the action images are classified using the action types identified by the action recognition model and arranged in chronological order in the action video. For example, the action types in the first 30 seconds of the first minute of the action video are respectively: squat, squat, squat... The action types in the last 30 seconds of the first minute are respectively: push-ups, push-ups, push-ups... In this way, the action images of each frame of the action video can be divided into multiple groups of action series images according to the action types in different time periods.

[0103] In this embodiment, after multiple groups of action series images are divided, action parameters are calculated for the action series images belonging to the same group. The action parameters include the joint point coordinates of the object to be measured, the movement trajectory of the joint point coordinates, the relative positions of adjacent joint points, the angles of the adjacent joint point connection line relative to the reference joint point connection line, etc. The action parameters of the action series images of the same group are respectively mapped to the coordinate axes, for example, the horizontal axis of the coordinate axis is time, and the vertical axis is the angle, or the horizontal axis of the coordinate axis is time, and the vertical axis is the relative distance of the joint points, so as to generate a parameter trend curve of the action parameters on the coordinate axis. By analyzing the peak points and trough points of the parameter trend curve and the slopes corresponding to these points, the extreme values ​​of the joint points in the movement process can be determined, such as For example, the maximum distance, the maximum angle value, the minimum distance, the minimum angle value, etc., so that the time corresponding to the trough point and the peak point can be accurately identified as the start time and the end time of the action, and then the action start frame and the action end frame of the reorganized action series image can be determined. In this way, when the limb relationship of the coordinates of each joint point of the object to be measured is subsequently calculated, the action angle can be identified for each group of images, and then the start and end of an action can be accurately identified, and based on the start and end of an action, the limb relationship can be accurately determined, and the movement of each limb relative to the reference limb in each action can be calculated, the end of the action can be accurately determined, and whether the object to be measured has completed the action can be accurately detected, thereby effectively improving the accuracy of action detection and counting.

[0104] In one embodiment, the step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type includes:

[0105] Extracting M consecutive test images from the action video in the time sequence at intervals of N frames, wherein N and M are positive integers greater than 2, and M is less than N;

[0106] Using the Yolo-Pose human posture estimation algorithm to identify the joint points of the object to be tested in each of the test images, and marking the joint points of each of the action images in the corresponding test image;

[0107] The test images marked with the joint points and the other action images are input into a pre-trained action recognition model in the time sequence, and the action recognition model is used to recognize the action of the object to be tested in the action image to obtain the action type.

[0108] In this embodiment, N frames are used as the interval for extracting test images, the time of the action images of N frames is related to the action type, and the time of the action images of N frames is close to or the same as the time of completing an action, so that each time the object to be tested performs an action, M frames of test images in the action can be extracted. For example, M is 30. Each time the object to be tested performs an action, the action images of the first 30 frames are extracted as the images to be tested.

[0109] In this embodiment, in order to improve the real-time recognition efficiency of the action recognition model, before the action image is input into the action recognition model, first, based on the approximate action completion time, continuous M frames of test images are extracted at intervals, so that M frames of test images are extracted at the beginning of each action, and then, the extracted test images are marked with joint points using the Yolo-Pose human posture estimation algorithm, for example, the name, coordinates, and limb of the joint point are marked. In this way, a marked test image is formed, and the marked test image and other unmarked action images are input into the action recognition model in the original time sequence for action recognition. Since the test images of the first M frames of each action are all marked with relevant node marks, the action recognition model can quickly and accurately recognize the action according to the trajectory of the joint point mark, and predict the action of the next NM frames of action images, and correct the action of the subsequent NM frames of action images according to the action recognized by the M frames, thereby effectively improving the action recognition accuracy, and in addition, effectively improving the action recognition efficiency. Compared with relying solely on the action recognition model to perform action recognition on all action images, using marked test images for auxiliary recognition can effectively improve recognition efficiency and accuracy.

[0110] It is worth mentioning that in this embodiment, joint point recognition and marking are not performed on all N frames of action images of each action, but only on the first M frames of images. The purpose of this is to avoid the inefficiency caused by the excessive amount of calculation of the Yolo-Pose human posture estimation algorithm for the recognition and marking of joint points. It should be understood that, relative to the increased amount of calculation and calculation load, the accuracy of action recognition is improved by using all N frames for joint point recognition and marking, and it does not reach the degree of the increased amount of calculation. That is to say, only using continuous M frames of images with intervals of N frames can improve the accuracy of action recognition to 100% or close to 100%. Using the Yolo-Pose human posture estimation algorithm to recognize and mark joint points for all N frames of images does not significantly improve the accuracy, but it significantly increases the amount of calculation, resulting in a decrease in the efficiency of the overall action type recognition. Therefore, in this embodiment, the Yolo-Pose human posture estimation algorithm is used to mark the joint points of the extracted M frame test images, which not only effectively improves the action recognition accuracy of the action recognition model, but also effectively improves the action recognition efficiency of the action recognition model.

[0111] In one embodiment, the step of inputting the test images marked with the joint points and other action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type includes:

[0112] The test images marked with the joint points and the other action images are input into a pre-trained action recognition model in the time sequence, and the action recognition model is used to recognize the action of the object to be tested in the action image to obtain the action type.

[0113] According to the marks of each of the test images, the posture of the object to be tested in the other action images is tracked, the key point area of ​​the object to be tested in the other action images is marked, and sparse processing is performed according to the marking results of the key point area, and the key pixels of the limbs of the object to be tested corresponding to the marks are retained to obtain a sparse image;

[0114] The test images and the sparse images marked with the joint points are input into a pre-trained action recognition model, and the action recognition model is used to recognize the action of the object to be tested in each of the test images and the action of the object to be tested in each of the sparse images to obtain the action type.

[0115] In this embodiment, sparse processing is used to remove redundant information in other action images and only retain key data points. Specifically, in this embodiment, the SIFT (Scale Invariant Feature Transform) algorithm is used, and the area around the marked joints of the test image is used as the key data points, and the background, clothing, etc. in the image are used as redundant information. According to the marking of the test image, posture tracking is performed on the action images of the NM frames between the first M frames of the test image and the second M frames of the test image, so as to achieve marking of the key point areas of the action images of the NM frames between the first M frames of the test image and the second M frames of the test image, and then the action images of the NM frames between the first M frames of the test image and the second M frames of the test image after the key point areas are marked are subjected to sparse processing to obtain an image that retains the key pixel data of the key limbs, thereby obtaining a sparse image. It is worth mentioning that the marking of the key point area here is different from the joint point marking in the above embodiment. In the above embodiment, the joint point marking requires extremely high precision and the joint point position needs to be determined very clearly. In this embodiment, it is only necessary to roughly mark the area around the joint point, because the larger the area retained later, the easier it is to identify the movement of the joint point. Moreover, if the area is too small, the difficulty of identification is increased, which is not conducive to improving efficiency. It should be understood that in this embodiment, it is only necessary to identify the area around the joint point, with the purpose of marking the approximate movement range of the joint point to facilitate the recognition of the action recognition model.

[0116] Through the above process, the amount of image data input to the action recognition model can be further reduced, the recognition efficiency of the model can be improved, and the detection accuracy of the action type can be further improved due to the reduction of noise interference. It is worth mentioning that since the joint points identified in the first M frames have a high enough recognition of the action, even if the subsequent NM frame images lose details, the resulting error is very small.

[0117] In one embodiment, the step of calculating the movement angle of a limb corresponding to the movement type in the movement trajectory based on the movement type to obtain the limb movement angle comprises:

[0118] Based on the action type, determining a plurality of reference limbs of the subject to be tested and a relative moving limb corresponding to each of the reference limbs;

[0119] According to the action type, each of the reference limbs and the relative moving limbs respectively corresponding to each of the reference limbs are tracked until the action image corresponding to the action type reaches a continuous preset number of frames, and a decomposed action trajectory is extracted from the motion trajectory;

[0120] The relative motion angle between the reference limb and the corresponding relative motion limb in the decomposed motion trajectory is calculated to obtain the limb motion angle.

[0121] In this embodiment, the reference limb is a reference object used to calculate the movement angle of the relatively moving limb, and the angle of the relatively moving limb is the angle between the reference limb and the reference. That is to say, the reference limb serves as the measurement reference for the angle of the limb, and the relatively moving limb is the limb that moves relative to the reference limb. In this embodiment, each action type corresponds to one or more reference limbs, and each relatively moving limb corresponds to a reference limb. In this embodiment, the reference limb corresponding to each action type and the relatively moving limb corresponding to the reference limb are pre-stored. Therefore, the reference limb and the corresponding relatively moving limb can be determined according to the determined action type, and then the coordinates of the reference limb and the corresponding relatively moving limb in multiple consecutive frames of action images are tracked. Specifically, the action images of the continuous preset number of frames are the multiple consecutive frames of the action images from the action start frame image to the action end frame image in each action series image of the same group.

[0122] In this embodiment, according to the action type, from the action start frame image to the action end frame image in the action series images of the same group, each of the reference limbs and the relative moving limbs corresponding to each of the reference limbs are tracked, until the action end frame image is tracked, the decomposed action trajectory is extracted from the motion trajectory, and the limb action angle change of the object to be tested in this section of the decomposed action trajectory is calculated. In this way, the limb action angle recognition of a complete action of the object to be tested can be completely performed, so as to accurately detect whether the object to be tested has completed an action completely, avoiding the problem of inaccurate action detection times caused by prematurely determining the end of the action due to inaccurate action completion detection. Through the above process, the inaccuracy of action number detection can be effectively avoided. In addition, since the number of frames corresponding to an action is determined, the interference of invalid actions and noise coordinates can be effectively avoided, and the detection accuracy is further improved.

[0123] In order to train a stable, reliable and accurate action recognition model, in one embodiment, the method further includes:

[0124] Get sample video;

[0125] Analyze the sample video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of sample joint coordinates of the sample object in the sample video;

[0126] Calculate the sample motion types and motion parameters of the sample object according to a preset number of sample joint coordinates;

[0127] Associating the sample action type and action parameters of the sample object with a plurality of sample images arranged in time series in the sample video;

[0128] The sample image and the sample action type and action parameters of the sample object after association are input into a support vector machine classification model, and the support vector machine classification model is trained to obtain the action recognition model.

[0129] In this embodiment, the sample object is a moving human body in the sample video, that is, a person whose action is detected. In this embodiment, the sample video includes videos from two camera positions.

[0130] In this embodiment, the Yolo-Pose human posture estimation algorithm is first used to extract the coordinates of 17 joint points of the human body from the sample video to obtain the sample joint coordinates. The Yolo-Pose human posture estimation algorithm is used to calculate the action type of the sample object based on the movement of a preset number of sample joint coordinates. The action type of the sample object is the sample action type. Specifically, the limb relationship of each sample joint coordinate is determined according to the movement range of the sample joint coordinates, and the movement trajectory of the limb is determined according to each sample joint coordinate determined by the limb relationship, so as to judge the action type of the sample object. In addition, the action parameters can also be calculated, and the action parameters include the movement angle, amplitude, etc. of the limbs of the sample object. Subsequently, the sample image of each frame is associated with the corresponding sample action type and action parameters. The association process is to mark the sample image. For example, the sample object in a sample image is marked as a squat, and the bending angle of the legs in the squat is 90 degrees. For example, the sample object in another sample image is marked as a push-up, and the bending angle of the arms in the push-up is 120 degrees. Subsequently, the associated sample images and the sample action types and action parameters of the sample objects are used as sample data sets, and the sample data sets are input into the support vector machine classification model for training, thereby obtaining the action recognition model.

[0131] Since the sample data set not only marks the action type corresponding to the sample image, but also marks the action parameters of the limb corresponding to the action type, it can enrich the accuracy of the sample data set of the support vector machine classification model, thereby effectively improving the robustness of the trained action recognition model.

[0132] In this embodiment, the sample data set is divided into a training set and a test set. The ratio usually adopted is that most of the samples (80%) of the data set are used for training, and the remaining part is used for testing (20%). The support vector machine model is trained using the training set to construct a support vector machine classifier. After the training is completed, the trained support vector machine model is tested using the test set. During the test, the sample features of the test set are input into the trained model, and the classification prediction is performed through the decision function of the model, and compared with the real label of the test set to evaluate the performance of the model.

[0133] The current results show:

[0134] - Of the samples in category 1 (squats), all 277 were correctly classified as squats.

[0135] - Of the samples in category 2 (pull-ups), all 257 were correctly classified as pull-ups.

[0136] - Of the samples in category 3 (push-ups), all 278 were correctly classified as push-ups.

[0137] - Of the samples of category 4 (flying birds), all 148 were correctly classified as flying birds.

[0138] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0139] Embodiment 2

[0140] In this embodiment, a motion detection device is provided, comprising:

[0141] An action video acquisition module, used to acquire action videos;

[0142] An action image parsing module is used to parse the action video to obtain a plurality of action images arranged in chronological order;

[0143] An action type recognition module, used for inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type;

[0144] A motion trajectory acquisition module is used to analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain the motion trajectory of the object to be measured in the action video;

[0145] A limb movement angle obtaining module, used to calculate the movement angle of the limb corresponding to the movement type in the movement trajectory based on the movement type, to obtain the limb movement angle;

[0146] A comparison module, used to detect whether the body movement angle meets the preset movement angle threshold corresponding to the movement type;

[0147] The determination module is used to determine that the action corresponding to the action type is completed when the limb action angle meets the preset action angle threshold corresponding to the action type.

[0148] The specific definition of the motion detection device can be found in the definition of the motion detection method above, which will not be repeated here. Each unit in the above-mentioned motion detection device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned units can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above-mentioned units.

[0149] Embodiment 3

[0150] In this embodiment, a computer device is provided, which may be a portable wearable device. In this embodiment, the computer device is a smart wristband. The internal structure diagram thereof may be as follows: Figure 2As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores a computer program, and the non-volatile storage medium is used to store the user's action video and action image. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other computer devices, such as connecting to a server, and the network interface is a wireless network interface that can access a mobile communication network and connect to a server. When the computer program is executed by the processor, a motion detection method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. In some embodiments, the input device includes a touch sensor, and the input device includes a photoelectric sensor, a skin electrical sensor, an inertial measurement unit (IMU), a temperature sensor and a sound sensor.

[0151] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0152] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in any of the above embodiments when executing the computer program.

[0153] Embodiment 4

[0154] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps described in any of the above embodiments are implemented.

[0155] Embodiment 5

[0156] In this embodiment, a computer program is provided. When the computer program is executed by a processor, the steps described in any of the above embodiments are implemented.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0158] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A motion detection method, characterized in that: include: Get action video; parsing the action video to obtain a plurality of action images arranged in chronological order; Inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type; The action video is analyzed using the Yolo-Pose human posture estimation algorithm to obtain a motion trajectory of the object to be measured in the action video; Based on the action type, calculating the movement angle of the limb corresponding to the action type in the motion trajectory to obtain the limb movement angle; Detecting whether the limb movement angle meets a preset movement angle threshold corresponding to the movement type; When the limb movement angle meets the preset movement angle threshold corresponding to the movement type, it is determined that the movement corresponding to the movement type is completed once.

2. The method according to claim 1, characterized in that The step of using the Yolo-Pose human posture estimation algorithm to analyze the action video to obtain the motion trajectory of the object to be measured in the action video includes: Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of joint point coordinates of the object to be measured in the action video; Detecting the motion range of each of the joint point coordinates in the action images of a preset number of consecutive frames, and determining the limb relationship of each of the joint point coordinates according to the motion range of each of the joint point coordinates; Based on the limb relationship, the limbs of the object to be measured and the coordinates of the joint points corresponding to each limb are determined, and the movement trajectory of the limbs of the object to be measured is determined according to the coordinates of the joint points corresponding to the limbs of the object to be measured.

3. The method according to claim 2, characterized in that The step of using the Yolo-Pose human posture estimation algorithm to analyze the action video to obtain a preset number of joint point coordinates of the object to be measured in the action video comprises: Analyze the action video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of original joint coordinates of the object to be measured in the action video; Determine a plurality of detection frames corresponding to the joints of the object to be detected, and extract the horizontal parameters and the vertical parameters of the detection frames; Calculating the horizontal relative ratio and the vertical relative ratio of each of the original joint coordinates to the starting coordinates of the detection frame by using the horizontal parameters and the vertical parameters of each of the original joint coordinates and the corresponding detection frame; According to the horizontal relative proportion and the vertical relative proportion, the original joint coordinates are subjected to normalized feature processing to obtain the joint point coordinates.

4. The method according to claim 2, characterized in that: The step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type comprises: Inputting each of the action images into a pre-trained action recognition model in the time sequence, using the action recognition model to recognize the action of the object to be tested in the action image, and obtaining the action type of the object to be tested in each of the action images; According to the action type of the object to be measured in each action image, grouping each action image in the time sequence to obtain multiple groups of action series images; Calculating the motion parameters of the object to be measured in each of the action series images belonging to the same group by using the Yolo-Pose human posture estimation algorithm, and generating a parameter trend curve based on the time sequence based on the motion parameters in each of the action series images belonging to the same group; Analyze the parameter trend curve to obtain the peak points, trough points and slopes corresponding to the peak points and trough points of the parameter trend curve, and determine the action start frame image and the action end frame image of the object to be measured in each of the action series images belonging to the same group based on the peak points, trough points and slopes corresponding to the peak points and trough points of the parameter trend curve; The step of detecting the motion range of each of the joint point coordinates in the action images of a preset number of consecutive frames, and determining the limb relationship of each of the joint point coordinates according to the motion range of each of the joint point coordinates comprises: Detect the movement range of each joint point coordinate in the action image of a continuous preset frame number, and determine the limb relationship of each joint point coordinate based on the movement range of each joint point coordinate, wherein the action image of a continuous preset frame number is a continuous plurality of frames of the action images from the action start frame image to the action end frame image in each action series image of the same group.

5. The method according to claim 1, characterized in that The step of inputting each of the action images into a pre-trained action recognition model in the time sequence, and using the action recognition model to recognize the action of the object to be tested in the action image to obtain the action type comprises: Extracting M consecutive test images from the action video in the time sequence at intervals of N frames, wherein N and M are positive integers greater than 2, and M is less than N; Using the Yolo-Pose human posture estimation algorithm to identify the joint points of the object to be tested in each of the test images, and marking the joint points of each of the action images in the corresponding test image; The test images marked with the joint points and other action images are input into a pre-trained action recognition model in the time sequence, and the action recognition model is used to recognize the action of the object to be tested in the action image to obtain the action type.

6. The method according to claim 1, characterized in that The step of calculating the movement angle of a limb corresponding to the movement type in the movement trajectory based on the movement type to obtain the limb movement angle comprises: Based on the action type, determining a plurality of reference limbs of the subject to be tested and a relative moving limb corresponding to each of the reference limbs; According to the action type, each of the reference limbs and the relative moving limbs respectively corresponding to each of the reference limbs are tracked until the action image corresponding to the action type reaches a continuous preset number of frames, and a decomposed action trajectory is extracted from the motion trajectory; The relative motion angle between the reference limb and the corresponding relative motion limb in the decomposed motion trajectory is calculated to obtain the limb motion angle.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Get sample video; Analyze the sample video using the Yolo-Pose human posture estimation algorithm to obtain a preset number of sample joint coordinates of the sample object in the sample video; Calculate the sample motion types and motion parameters of the sample object according to a preset number of sample joint coordinates; Associating the sample action type and action parameters of the sample object with a plurality of sample images arranged in time series in the sample video; The sample image and the sample action type and action parameters of the sample object after association are input into a support vector machine classification model, and the support vector machine classification model is trained to obtain the action recognition model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images