Video image motion counting method based on human key point action sequence matching

Abstract

The application discloses a video image motion counting method based on human key point action sequence matching. A human body moves in the camera field of view, the video image of the human body movement is collected through the camera, the type and counting of the movement are obtained through analysis and processing of the video image, specifically, motion counting based on action sequence splitting, different templates are established to match and identify each motion action, and adaptive distribution is performed in different computing devices. The application has higher counting accuracy and stability on a small model with relatively low key point detection accuracy, meets a motion scene with higher detection speed requirement, has the advantages of fast running speed, stable action matching, high accuracy, high counting accuracy, and suitability for running on different deployment devices.

Description

Technical Field

[0001] This invention relates to a video image motion recognition method in the field of sports counting, and more particularly to a video image motion counting method based on human key point action sequence matching. Background Technology

[0002] In fields such as physical education exams, physical fitness tests, exercise detection, physical fitness assessment, and fitness training, key point detection based on visual algorithms can acquire multiple key points of the human body. By judging the relative positional relationship of key points, it can reflect different movement states of athletes. This allows for further use of algorithms to perform functions such as action matching degree judgment and motion counting, improving the accuracy and stability of motion counting and action judgment, providing better and more precise guidance for athletes' exercise and fitness, and saving the cost of manual statistics and counting.

[0003] The sports counting and motion matching method based on human key points generally uses a camera to acquire real-time video footage of human movement, and then uses a deep learning human key point detection algorithm to identify human key point information (such as nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, left and right ankles, etc.) frame by frame. Then, based on the transformation patterns of key points in space and time, counting rules or motion state matching methods are constructed.

[0004] Traditional motion counting or action matching methods typically formulate rules based solely on the coordinates of key points on the human body. For example, counting pull-ups generally follows these rules: a pull-up is considered a stretching state if the distance between the wrist and shoulder joints is less than a certain threshold, and a downward state if the distance is greater. Each detection of a stretching or hanging state results in one count. This is the general workflow for motion counting based on key points. Traditional counting methods usually employ only one method for action matching. If the key points required for action judgment are missing or abnormal, the entire counting and judgment chain will fail. They rely heavily on the absolute accuracy of key point detection and lack robustness to abnormal or missing key point information. Furthermore, traditional key point-based motion counting methods do not consider the differences in detection rates (frame rates) across different devices, leading to poor stability and robustness of the formulated counting or action matching rules, making them unsuitable for devices with varying computing power. Summary of the Invention

[0005] Based on the above analysis, the present invention provides a video image motion counting method based on human key point action sequence matching.

[0006] This invention can be applied to sports activity counting, sports test examinations, fitness exercises, and physical fitness assessments. The method is suitable for counting sports activities such as rope skipping, squats, push-ups, planks, jumping jacks, sit-and-reach, running in place, and sit-ups.

[0007] The purpose of this invention is to address the shortcomings of existing technologies by establishing the following technical solution:

[0008] The human body moves within the camera's field of view. The camera captures video images of the human body's movements. The type and count of the movements are obtained through analysis and processing of the video images. Specifically, the movement count is based on the action sequence segmentation. Different templates are established to match and identify each movement action, and the process is adaptively allocated and run on different computing devices.

[0009] The method breaks down the motion counting process into multiple sub-motion matching processes, specifically:

[0010] First, each motion type is divided into a combination consisting of multiple consecutive sub-motion states. Different motion types are divided into different combinations, and a template is created for each sub-motion state.

[0011] Then, the complete video images are matched sequentially using templates for each different sub-action state to obtain a sequence of sub-action states according to the temporal order of the video images;

[0012] Finally, the completion degree of the sub-action is extracted based on the sub-action state sequence. The completion degree of each sub-action state is counted, and the total motion count is calculated by combining the overall motion type under the combination of multiple sub-action states.

[0013] First, the key point information of each frame in the video is obtained. Then, the key point information at the current moment is filtered based on the detection confidence, the rationality of human body structure, and the relationship of neighboring key points in the time dimension to remove abnormal or unstable key points. Finally, the video images are matched with templates in time sequence.

[0014] Furthermore, for each sub-action state, different action matching templates are designed and established from different dimensions and using different action matching methods. The matching judgment is made by fusing multiple action matching templates, and one of the following methods is selected:

[0015] Method 1: Sort the action matching templates according to their stability and select the one with the highest priority for matching.

[0016] Method 2: Use a voting mechanism to match all action matching templates separately and determine whether they match, and take the majority of the results as the outcome.

[0017] The method runs on different computing devices and automatically calculates and allocates the frame rate for sub-action state matching and judgment according to the different computing power of the computing devices. The stronger the computing power of the computing device, the more frame rate of the judgment image is allocated for each sub-action state matching and judgment A, thus adapting to computing devices with different computing power levels.

[0018] This invention employs a motion counting method based on sub-action sequence decomposition, which breaks down the motion counting process into multiple sub-action matching processes. Motion counting is performed based on the completion degree of sub-action matching and state transition status. A timeout mechanism and a multiple confirmation voting mechanism are designed to reduce action matching misjudgments.

[0019] This invention employs a multi-dimensional action matching method, designing rules from different dimensions to confirm action matching. In cases of anomalies or occlusions in some key points, the corresponding dimension's judgment path is automatically blocked, and the action matching judgment is performed using the intact dimension's path. This enhances the robustness and stability of action matching.

[0020] This invention employs an adaptive rule-based counting method that adapts to varying detection rates. Depending on the detection speed, the number of action matching confirmations and the maximum response time per detection cycle in the counting rules are adaptively adjusted.

[0021] The technical solution of this invention can be summarized as follows:

[0022] 1. A motion counting method based on sub-action sequence decomposition is proposed. The motion counting process is broken down into multiple sub-action matching processes, and motion counting is performed based on the sub-action matching completion degree and state transition status.

[0023] 2. A multi-dimensional action matching method is proposed, which designs rules from different dimensions to confirm action matching. In the event of anomalies or occlusions of some key points, the corresponding dimension judgment route is automatically blocked, and the action matching judgment is performed using the dimension route that is not missing. This enhances the robustness and stability of action matching.

[0024] 3. An adaptive detection rate-based rule counting method is proposed. Based on different device computing power, the frame rate allocated for action state matching and determination A is automatically calculated, adapting to devices with varying computing power levels. Depending on the detection speed, the number of action matching confirmations in the counting rules is adaptively adjusted, improving the stability of the determination from a time perspective.

[0025] The beneficial effects of this invention are:

[0026] The motion counting method of the present invention, which splits the motion counting process into multiple sub-action matching processes, performs motion counting based on the completion degree of sub-action matching and state transition. It is applicable to different sports counting projects, and other motion counting processes can be easily designed according to this method.

[0027] The multi-dimensional action matching method of the present invention still has good robustness in the case of missing or abnormal key points, further improving the counting accuracy.

[0028] The adaptive detection rate change frame rate allocation method of the present invention ensures that the entire rule judgment process does not fail under different frame rates, and transforms computing power speed into counting stability.

[0029] The motion counting method of this invention has the characteristics of high counting robustness and high accuracy, and still maintains a high accuracy even when the detection of key points is relatively low or there are some anomalies. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method of the present invention.

[0031] Figure 2 This is a diagram illustrating the next count determination after the timeout period.

[0032] Figure 3 This is a schematic diagram illustrating the matching of action states in an action sequence.

[0033] Figure 4 This diagram illustrates the situation when the action state matching judgment method fails. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 1 As shown, the specific implementation process of motion counting in this invention is as follows:

[0036] Before a user engages in exercise and the camera captures video, the user needs to input the type of exercise. The system directly accepts the input type of exercise, rather than determining it from the video.

[0037] The user inputs the type of movement they want to perform into the system, and the human body moves within the camera's field of view. The camera captures video images of the human body's movement, and the system analyzes and processes these video images to obtain movement counts. Specifically, the movement counts are based on the action sequence segmentation. Different templates are established to match and identify each movement action, and the system is adaptively allocated to run on different computing devices.

[0038] The specific implementation involves breaking down the motion counting process into multiple sub-motion matching processes, specifically:

[0039] First, each motion type is divided into a combination consisting of multiple consecutive sub-motion states. Different motion types are divided into different combinations, and a template is created for each sub-motion state.

[0040] Then, the complete video images are matched sequentially using templates for each different sub-action state to obtain a sequence of sub-action states according to the temporal order of the video images;

[0041] Finally, the completion degree of the sub-action is extracted based on the sub-action state sequence. The completion degree of each sub-action state is counted, and the total motion count is calculated by combining the overall motion type under the combination of multiple sub-action states.

[0042] This breaks down the motion counting process SP into a matching process of n sub-motion states SP = [A1, A2, A3, ..., A n ], where A1, A2, A3, ..., A n Each represents one of the n sub-action states.

[0043] The table below shows some examples of motion counting action breakdown.

[0044] Types of sports Sub-action sequence matching and splitting Pull-ups [Stretched state, Released state] squat [Standing position, lowering position, squatting position, standing position] Jumping Jacks [Jump-in state, Jump-out state] .... ......

[0045] Then, according to the sub-action state sequence [A1, A2, A3, ..., A n The state-action matching judgment is performed sequentially. When all action states under a motion type have been matched in sequence, one motion type count is completed. If a sub-action state does not match, the subsequent actions are not judged, and the next motion type count judgment is performed directly.

[0046] The timeout period TN is set according to the action type. If the timeout period is exceeded, the count will automatically increase for the next judgment. The specific process is as follows: Figure 2 As shown.

[0047] First, the key point information of each frame in the video is obtained. Then, the key point information at the current moment is filtered based on the detection confidence, the rationality of human body structure, and the relationship of neighboring key points in the time dimension to remove abnormal or unstable key points. Finally, the video images are matched with templates in time sequence.

[0048] like Figure 3 As shown, different action matching templates are designed and established for each sub-action state from different dimensions and using different action matching methods. When there are multiple action matching templates for a sub-action state, the matching judgment is made by fusing multiple action matching templates to enhance the stability of action matching. One of the following methods is selected:

[0049] Method 1: Sort the action matching templates according to their stability and select the one with the highest priority for matching.

[0050] Method 2: Use a voting mechanism to match all action matching templates separately and determine whether they match, and take the majority of the results as the outcome.

[0051] In this way, when the action state matching A fails due to missing key points causing the template of the judgment method to become invalid, templates of other judgment methods can still be used for judgment, providing more judgment paths and stronger stability of motion counting, such as... Figure 4 As shown.

[0052] The method runs on different computing devices and automatically calculates and allocates the frame rate for sub-action state matching and judgment according to the different computing power of the computing devices. The stronger the computing power of the computing device, the more frame rate of the judgment image is allocated for each sub-action state matching and judgment A, thus adapting to computing devices with different computing power levels.

[0053] This invention also establishes a rule-based counting method that adapts to changes in detection rate. Specifically, the number of sub-action state matching judgments (A) within an action counting inspection cycle is fixed. The stronger the computing power of the device, the faster the keypoint detection speed, and the more frame rates of the judgment images allocated for each sub-action state matching judgment (A). This makes it applicable to different computing devices, and higher-performance computing devices provide more keypoint information for judgment (A), resulting in higher accuracy in action matching and counting.

[0054] An implementation example involves deploying the algorithm of this invention on mobile phones, computers, and edge computing devices. First, the device's built-in camera (e.g., a mobile phone) or an external network camera or USB camera is used to obtain key point detection information using a key point detection model (applicable to different key point models). Then, the algorithm of this invention is used to perform motion counting, including but not limited to: squat counting, jumping jack counting, rope skipping counting, push-up counting, sit-up counting, etc.

[0055] In practical implementation, this invention demonstrates high counting accuracy and stability even on small models with relatively low keypoint detection accuracy. It also meets the requirements for motion scenarios with high detection speeds. Compared to ordinary motion counting algorithms based on human keypoints, this invention offers advantages such as faster running speed, higher stability and accuracy in motion matching, higher counting accuracy, and applicability to various deployment devices.

Claims

1. A video image motion counting method based on human keypoint action sequence matching, characterized in that: Human body moves in the camera field of view, the video image of human body movement is collected by the camera, the type and count of movement are obtained by analyzing and processing the video image, specifically, the movement count based on action sequence splitting, different templates are established to match and identify each movement action, and adaptive distribution is run in different computing devices; The method will split the movement counting process into multiple sub-action matching processes, specifically: First, each movement type is divided into a combination composed of multiple continuous sub-action states, different movement types are divided into different combinations, and templates are established for each sub-action state; Then, the complete video image is matched with the templates of each different sub-action state in time sequence to obtain the sub-action state sequence under the video image time sequence; Finally, the sub-action completion degree is extracted according to the sub-action state sequence, the single sub-action state is counted according to the sub-action completion degree, and the total movement count is obtained according to the overall situation of the movement type under the combination of multiple sub-action states; First, the key point information of each frame image in the video is obtained, then the key point information at the current time is filtered according to the detection confidence, human body structure rationality and relationship of adjacent key points in time dimension to remove abnormal or unstable key points, and then the video image is matched with the template in time sequence; And different action matching templates are designed and established for each sub-action state from different dimensions and using different action matching methods, and the following methods are selected: Method 1: The action matching templates are sorted according to the priority according to the judgment stability, and the highest priority is selected for matching judgment; Method 2: A voting mechanism is adopted, all action matching templates are matched and judged respectively, and the majority decision is taken as the result; In this way, when the action state is missing due to key point loss, the template of the judgment method is invalid, and other templates of the judgment method can be used for judgment; The method runs in different computing devices, and according to the different computing power of the computing devices, the frame rate for sub-action state matching judgment is automatically calculated and distributed, the stronger the computing power of the computing device, the more the frame rate distribution for each sub-action state matching judgment A, and the adaptive computing device with different computing power.

Images