Method and device for counting opening and closing jumps based on human key points, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and provides a body key point-based jumping jack counting method and device, equipment and a medium. A video stream in a jumping jack movement process of a human body is monitored to obtain a movement video frame picture, a body posture detection model is used to obtain the coordinates of a plurality of body key points in the movement video frame picture, the coordinates of the plurality of body key points in the movement video frame picture are used in combination with complete and reasonable starting and ending action judgment standards of the jumping jack to judge the process of the jumping jack action, and the valid starting action of the jumping jack and the valid ending action of the jumping jack are detected, so that the accuracy of the jumping jack calculation in AI movement detection is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for counting jumping jacks based on key points on the human body. Background Technology

[0002] Home workouts have become increasingly popular, indicating that more and more people are paying attention to their health. In recent years, the number of people enthusiastic about fitness has surged, and correct exercise posture is paramount. Incorrect posture during exercise not only fails to provide the desired fitness benefits but can also cause long-term harm. Previously, people could correct and train their posture under the guidance of fitness instructors. Now, there is a pressing need for supervision and prompts during the exercise process, leading to the emergence of AI-powered motion detection.

[0003] Currently, the algorithms used in AI motion detection apps are not very mature, leading to issues such as: counts failing to start when following the instructions, inaccurate posture detection, and inaccurate counting. This indicates a need for more precise algorithms to improve the accuracy of AI detection.

[0004] Therefore, developing an opening and closing jump counting method that can improve the accuracy of AI detection is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method, device, equipment and medium for counting opening and closing jumps based on key points of the human body, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0006] Firstly, a jumping jack counting method based on key points of the human body is provided, including:

[0007] Monitor the video stream during the human jumping jacks exercise to obtain the first motion video frame image;

[0008] The coordinates of several key human body points in the first motion video frame image are obtained using a human pose detection model and used as the coordinates of the first key point.

[0009] Based on the coordinates of the first key point, the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame is compared frame by frame until the first target key point and the second target key point in a certain frame of the first motion video frame satisfy the first positional relationship, which is recorded as the effective starting action of the jumping jack.

[0010] After detecting a valid starting motion of the jumping jacks, continue to monitor the video stream during the human jumping jacks motion to obtain the second motion video frame image;

[0011] The coordinates of several key human body points in the second motion video frame image are obtained using the human body posture detection model, and are used as the coordinates of the second key points.

[0012] Based on the coordinates of the second key point, the positional relationship between the first target key point and the preset position in each frame of the second motion video frame is compared frame by frame until the first target key point and the preset position in a certain frame of the second motion video frame satisfy the second positional relationship, which is recorded as the effective end action of the jumping jack.

[0013] Increment the jumping jack count by 1, and re-execute the step of monitoring the video stream during the human jumping jack movement to obtain the first motion video frame image until the preset stopping condition is met.

[0014] Secondly, a jumping jack counting device based on key points of the human body is provided, including:

[0015] The first image acquisition module is used to monitor the video stream during the human jumping jacks exercise to acquire the first motion video frame image;

[0016] The first coordinate acquisition module is used to acquire the coordinates of several human key points in the first motion video frame image using a human pose detection model, and use them as the coordinates of the first key points.

[0017] The first action judgment unit is used to compare the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame image according to the coordinates of the first key point, until the first target key point and the second target key point in a certain frame of the first motion video frame image satisfy the first positional relationship, and then record it as a valid starting action of the jumping jack.

[0018] The second image acquisition module is used to continue monitoring the video stream during the human jumping jack movement after detecting the effective starting action of the jumping jack, so as to acquire the second motion video frame image.

[0019] The second coordinate acquisition module is used to acquire the coordinates of several key human body points in the second motion video frame image using the human body posture detection model, and use them as the coordinates of the second key points.

[0020] The second action judgment unit is used to compare the positional relationship between the first target key point and the preset position in each frame of the second motion video frame image according to the coordinates of the second key point, until the first target key point and the preset position in a certain frame of the second motion video frame image satisfy the second positional relationship, which is recorded as the effective end action of the jumping jack.

[0021] The counting module is used to increment the jumping jack count by 1 and re-execute the step of monitoring the video stream during the human jumping jack movement to obtain the first motion video frame image until the preset stopping condition is met.

[0022] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described jumping jack counting method based on human body key points.

[0023] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described jumping counting method based on human body key points.

[0024] The aforementioned method, apparatus, device, and medium for counting jumping jacks based on human body key points acquire a first motion video frame image by monitoring the video stream during the human body's jumping jack movement; it uses a human posture detection model to obtain the coordinates of several human body key points in the first motion video frame image, which are then used as the coordinates of the first key points; based on the coordinates of the first key points, it compares the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame image frame by frame, until the first target key point and the second target key point in a certain frame of the first motion video frame image satisfy the first positional relationship, which is then recorded as a valid starting action of the jumping jack; after detecting a valid starting action of the jumping jack, it continues to monitor the human body's jumping jack movement. The video stream during the jumping jack movement is used to obtain a second motion video frame image. The coordinates of several key human points in the second motion video frame image are obtained using the human posture detection model, serving as the second key point coordinates. Based on the second key point coordinates, the positional relationship between the first target key point and a preset position in each frame of the second motion video frame image is compared frame by frame until the first target key point and the preset position in a certain frame of the second motion video frame image satisfy the second positional relationship, at which point it is recorded as a valid end action of the jumping jack. The jumping jack count is incremented by 1, and the steps of monitoring the video stream during the human jumping jack movement to obtain a first motion video frame image are repeated until a preset stopping condition is met. Using this invention, the jumping jack action can be correctly identified, improving the accuracy of jumping jack calculation in AI-based motion detection. Attached Figure Description

[0025] Figure 1 A schematic diagram of an application environment for the opening and closing jump counting method based on human body key points of the present invention is shown.

[0026] Figure 2 A flowchart of the opening and closing jump counting method based on human body key points of the present invention is shown.

[0027] Figure 3 A sub-flowchart of step S30 in a specific embodiment is shown.

[0028] Figure 4 A sub-flowchart of step S40 in a specific embodiment is shown.

[0029] Figure 5 A functional block diagram of the opening and closing jump counting device based on human body key points of the present invention is shown.

[0030] Figure 6 A schematic diagram of an electronic device based on the opening and closing jump counting method of the present invention is shown. Detailed Implementation

[0031] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0032] Please see Figure 1-6 It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0033] The jumping jack counting method based on human key points provided in this invention can be applied to, for example... Figure 1In the illustrated application environment, the jumping jack counting application 200 based on human key points is installed on a mobile terminal 100, which may be, for example, a mobile phone or a tablet computer. The jumping jack counting application 200 requests camera access from the mobile terminal 100 via a method provided by the client. After successfully obtaining access, it monitors the video stream during the human jumping jack movement to obtain a first motion video frame image. It then uses a human posture detection model to obtain the coordinates of several human key points in the first motion video frame image. Based on the coordinates of these key points, it compares the positional relationship between the wrist and ear in each frame of the first motion video frame image until the wrist is positioned above the ear in a certain frame, at which point it is recorded as a valid starting action for the jumping jack. After detecting a valid starting action of a jumping jack, the video stream during the human jumping jack movement is continuously monitored to obtain a second motion video frame image. The coordinates of several key human points in the second motion video frame image are obtained using the human posture detection model. Based on the coordinates of the second key points, the positional relationship between the first target key point and a preset position in each frame of the second motion video frame image is compared frame by frame until the first target key point and the preset position in a certain frame of the second motion video frame image satisfy a second positional relationship, at which point it is recorded as a valid ending action of the jumping jack. The jumping jack count is incremented by 1, and the above steps are repeated until the jumping jack movement ends. Using this invention, the jumping jack action can be correctly identified, improving the accuracy of jumping jack calculation in AI-based motion detection.

[0034] When performing jumping jack counting based on human keypoints, it is necessary to first request camera permissions from the mobile terminal 100 through the method provided by the jumping jack counting application 200 client. After successfully obtaining permissions, the video stream is monitored using navigator.mediaDevices.getUserMedia. An image is drawn every 25 milliseconds (which can be configured appropriately according to the performance of the mobile terminal 100) using a canvas, thereby acquiring motion video frame images or pre-processed video frame images. A human pose detection model is loaded, which is used to detect motion video frame images or pre-processed video frame images, accurately detecting several human keypoints, thereby obtaining the coordinates of these keypoints, and drawing lines and points using a custom method. In this embodiment, the MoveNet motion detection model will be used as an example for illustration. Of course, the BlazePose model or PoseNet model can also be used.

[0035] The MoveNet motion detection model can accurately identify the coordinates of key human body points, such as the left and right eyes, nose, mouth, left and right ears, left and right shoulder joints, left and right elbows, left and right wrists, hip joints, left and right knee joints, and left and right ankles—a total of 17 key human body points. Once the coordinates of these points are obtained, the user's current position and posture can be determined. AI motion detection involves capturing the posture of movements using a camera and applying appropriate algorithms to analyze and calculate the captured movements, such as angle determination, voice prompts for incorrect movements, and red highlighting.

[0036] In this embodiment, a Canvas window coordinate system is used, including an x-axis and a y-axis. The intersection of the two axes is the origin (0, 0). Positive values ​​are defined along the x-axis to the right and downwards along the y-axis. Of course, other coordinate systems, such as a Cartesian coordinate system, can be used in other embodiments.

[0037] Before starting the jumping jack exercise, the user undergoes a readiness check. This check verifies whether the user's feet are together. If the readiness requirement is not met, a voice prompt will be given. The condition for feet to be together is that the distance between the left and right ankles is less than the product of the distance between the left and right shoulder joints and a preset ratio, and this is maintained for a certain period of time. The preset ratio is 0.5-1, for example, 0.5, 0.7, or 0.9. After passing the check, the coordinates of the wrists in the stable state (feet together) are recorded. The y-axis coordinate value of the wrist minus a preset pixel value is used as the threshold for the downward-facing hand position during the jumping jack exercise, and the position corresponding to the threshold is used as the preset position.

[0038] In other words, the method for obtaining the preset position includes: monitoring the video stream during the preparation phase of a human jumping jack to obtain preparation video frame images; using a human posture detection model to obtain the coordinates of several key human points in the preparation video frame images; based on the coordinates of the key human points in the preparation video frame images, determining frame by frame whether the distance between the left and right ankles in each frame of the preparation video frame images meets the condition of feet together; when the distance between the left and right ankles in a certain frame of the preparation video frame images meets the condition of feet together, recording the coordinates of the wrist in that preparation video frame image, and subtracting the preset pixel value from the y-axis coordinate value of the (left or right) wrist or elbow (i.e., the first target key point) in the preparation video frame image to obtain the y-axis coordinate value of the preset position. As an example, the preset pixel value can be reasonably set according to the different sizes of the canvas and / or the screen resolution of the mobile terminal 100. As an example, the preset pixel value is between 25 and 35, for example, 30.

[0039] To improve the accuracy of jumping jack counting, reduce the counting judgment of jumping jack movements for accurate counting, and simplify the entire technical process, this application decomposes the jumping jack process into two action units as a complete jumping jack action judgment. The first action unit is the take-off process with feet open and arms upward, and the second action unit is the judgment of the landing state with arms down and feet together.

[0040] The method for determining the first action unit is that the wrist is above the ear, indicating an upward trend. That is, the y-axis coordinate of the wrist is less than the y-axis coordinate of the ear (the coordinate of the left wrist is less than the coordinate of the left ear and the coordinate of the right wrist is less than the coordinate of the right ear). This is used as the starting threshold for a valid action and is recorded as the valid starting action of the jumping jack. If this valid threshold is not reached, the action is considered invalid.

[0041] The second action unit is the judgment of the falling state with both hands down and feet together. The judgment method for this unit is: when the wrist is below the preset position during the falling process, that is, when the y-axis coordinate value of the wrist reaches the judgment threshold of the jumping jack hand downward state calculated in the preparation stage, it is considered as a valid end action of the jumping jack.

[0042] Please see Figure 2 As shown, Figure 2 This is a flowchart illustrating a jumping jack counting method based on human body key points provided in an embodiment of the present invention. The jumping jack counting method based on human body key points in this embodiment of the present invention includes the following steps:

[0043] Step S10: Monitor the video stream during the human jumping jacks exercise to obtain the first motion video frame image;

[0044] Step S20: Use the human pose detection model to obtain the coordinates of several human key points in the first motion video frame image, and use them as the coordinates of the first key point;

[0045] Step S30: Based on the coordinates of the first key point, compare the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame image until the first target key point and the second target key point in a certain frame of the first motion video frame image satisfy the first positional relationship, and record it as a valid starting action of the jumping jack.

[0046] Step S40: After detecting a valid starting motion of the jumping jacks, continue to monitor the video stream during the human jumping jacks motion to obtain the second motion video frame image;

[0047] Step S50: Use the human pose detection model to obtain the coordinates of several human key points in the second motion video frame image, and use them as the coordinates of the second key points;

[0048] Step S60: Based on the coordinates of the second key point, compare the positional relationship between the first target key point and the preset position in each frame of the second motion video frame image until the first target key point and the preset position in a certain frame of the second motion video frame image satisfy the second positional relationship, and record it as the effective end action of the jumping jack.

[0049] Step S70: Increment the jumping jack count by 1, and re-execute the step of monitoring the video stream during the human jumping jack movement to obtain the first motion video frame image until the preset stopping condition is met.

[0050] First, perform step S10: monitor the video stream during the human jumping jacks exercise to obtain the first motion video frame image.

[0051] Once the movement begins, the readiness detection passes, and the relevant judgment threshold is obtained, the jumping jack movement can officially begin. At this time, the video stream during the jumping jack movement is monitored through navigator.mediaDevices.getUserMedia. An image is drawn once every preset time interval (which can be reasonably configured according to the performance of the mobile terminal, such as 25 milliseconds) through the canvas, thereby obtaining the motion video frame image, which is defined as the first motion video frame image.

[0052] Next, step S20 is executed: the coordinates of several human key points in the first motion video frame image are obtained using the human pose detection model, and used as the coordinates of the first key point.

[0053] The MoveNet motion detection model is used to detect the first motion video frame image to obtain the coordinates of several human key points in the first motion video frame image. The human key points may include the left and right wrists, left and right ears, left and right shoulder joints, left and right ankles, left and right elbows, left and right eyes, etc.

[0054] Next, step S30: Based on the coordinates of the first key point, compare the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame image, until the first target key point and the second target key point in a certain frame of the first motion video frame image satisfy the first positional relationship, and record it as a valid starting action of the jumping jack.

[0055] In this embodiment, the first target key point includes the wrist or elbow, and the second target key point includes the ear, eye, or shoulder joint. The first positional relationship is that the first target key point is located above the second target key point. Therefore, when determining the effective starting action of the jumping jack, the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame can be compared according to the coordinates of the first key point until the first target key point in a certain frame of the first motion video frame is located above the second target key point, which is recorded as the effective starting action of the jumping jack.

[0056] It is understood that in other embodiments, the first target key point may also be other suitable human body key points other than the wrist or elbow, and the second target key point may be other suitable human body key points other than the eyes, shoulder joint, etc. Depending on the selection of the first target key point and the second target key point, the first positional relationship can be configured as needed.

[0057] like Figure 3 As shown, in a specific embodiment, the first target key point includes the left and right wrists, and the second target key point includes the left and right ears. Based on the coordinates of several human body key points in the first motion video frame image, the positional relationship between the wrists and ears in each frame of the first motion video frame image is compared frame by frame until the wrists in a certain frame of the first motion video frame image are located above the ears, which is recorded as a valid starting action for the jumping jack. This may further include:

[0058] Step S31: Obtain the coordinates of the left and right wrists and the left and right ears in the first motion video frame image of the current frame;

[0059] Step S32: Determine whether the coordinates of the left and right wrists and the coordinates of the left and right ears in the first motion video frame image of the current frame simultaneously meet the following conditions: the y-axis coordinate value of the left wrist is less than the y-axis coordinate value of the left ear, and the y-axis coordinate value of the right wrist is less than the y-axis coordinate value of the right ear.

[0060] Step S33: When the condition is met, it means that the wrist in the first motion video frame image of the current frame is above the ear, which is recorded as the effective starting action of the jumping jack;

[0061] Step S34: If the condition is not met, the first motion video frame image of the next frame is used as the first motion video frame image of the current frame, and the step of obtaining the coordinates of the left and right wrists and the left and right ears in the first motion video frame image of the current frame is returned (i.e., step S31).

[0062] After the effective starting action of the jumping jack is detected through steps S10-S30, it means that the user has completed the jumping action with both feet open and arms pointing upward. Next, it is necessary to continue monitoring the video stream during the human jumping jack movement to detect the effective ending action of the jumping jack, corresponding to steps S40-S60.

[0063] Next, step S40 is executed: after detecting a valid starting action of the jumping jack, the video stream during the human jumping jack movement is monitored to obtain the second motion video frame image.

[0064] Continue monitoring the video stream during the human jumping jacks exercise, and draw an image at preset time intervals using a canvas to obtain motion video frame images, which are defined as the second motion video frame images.

[0065] Step S50: Use the human posture detection model to obtain the coordinates of several human key points in the second motion video frame image, and use them as the coordinates of the second key points.

[0066] Specifically, the MoveNet motion detection model can be used to detect the acquired second motion video frame image to obtain the coordinates of several human key points in the second motion video frame image. The human key points may include the left and right wrists, left and right ears, left and right shoulder joints, left and right ankles, left and right elbows, left and right eyes, etc.

[0067] Step S60: Based on the coordinates of the second key point, compare the positional relationship between the first target key point and the preset position in each frame of the second motion video frame until the first target key point and the preset position in a certain frame of the second motion video frame satisfy the second positional relationship, and record it as the effective end action of the jumping jack.

[0068] In this embodiment, the first target key point includes the wrist or elbow, and the second positional relationship is that the first target key point is located below the preset position. Therefore, when determining the effective end of the jumping jack action, the positional relationship between the first target key point and the preset position in each frame of the second motion video frame is compared frame by frame based on the coordinates of several human body key points in the second motion video frame image, until the first target key point in a certain frame of the second motion video frame is located below the preset position, at which point it is recorded as the effective end of the jumping jack action. It is understood that in other embodiments, the first target key point and the second positional relationship can also be configured as needed.

[0069] like Figure 4As shown, in a specific embodiment, the first target key point includes the left and right wrists. Based on the coordinates of several human body key points in the second motion video frame image, the positional relationship between the wrist and the preset position in each frame of the second motion video frame image is compared frame by frame until the wrist in a certain frame of the second motion video frame image is below the preset position, which is recorded as the effective end action of the jumping jack. It may further include:

[0070] Step S61: Obtain the coordinates of the left and right wrists in the second motion video frame image of the current frame;

[0071] Step S62: Determine whether the coordinates of the left and right wrists in the second motion video frame image of the current frame and the coordinates of the preset position simultaneously meet the following conditions: the y-axis coordinate value of the left wrist is greater than the y-axis coordinate value of the preset position, and the y-axis coordinate value of the right wrist is greater than the y-axis coordinate value of the preset position.

[0072] Step S63: When the condition is met, it means that the wrist in the second motion video frame image of the current frame is below the preset position, and it is recorded as the effective end action of the jumping jack.

[0073] Step S64: If the condition is not met, the next frame of the second motion video frame image is used as the current frame of the second motion video frame image, and the process returns to the step of obtaining the coordinates of the left and right wrists in the current frame of the second motion video frame image (i.e., step 61).

[0074] Finally, execute step S70: increment the jumping jack count by 1, and re-execute the step of monitoring the video stream during the human jumping jack movement to obtain the first motion video frame image until the preset stopping condition is met.

[0075] When a valid start and end action of a jumping jack are detected consecutively, a complete jumping jack movement is considered to be completed. The counter is incremented by 1, and the process returns to step S10. Steps S10-S70 are then repeated until the jumping jack movement ends (a preset stop condition is met). The jumping jack movement can end upon receiving a human-generated end command (e.g., clicking "End" on the application interface or using voice control), or by clicking "End" on the application interface, or by not detecting a valid start and / or end action of a jumping jack within a preset time period.

[0076] It should be noted that, to reinforce the requirement for standard posture during exercise, the jumping jack counting method based on human body key points further includes detecting non-standard movements based on the coordinates of several human body key points in the second motion video frame image; scoring the jumping jack movements based on the non-standard movement detection results, and marking the human body key points corresponding to non-standard movements (e.g., connecting them with red lines) and displaying them on the display interface to remind the user. For example, the non-standard movement detection could be, for instance, detection of reversed hand and foot positions, excessive arm bending, etc.

[0077] The process of detecting the correctness of posture during exercise is illustrated by taking the example of detecting the reverse direction of the hands and feet and excessive bending of the arms during a jump with the feet open and the arms pointing upward.

[0078] Hand and foot reversal detection: After a valid starting action of the jumping jack is detected, the system can determine whether the distance between the left and right ankles, the distance between the left and right shoulder joints, and the distance between the left and right wrists when clapping hands meet the following conditions: the distance between the left and right ankles < the distance between the left and right shoulder joints < the distance between the left and right wrists. If the conditions are met, the action is considered standard. If the conditions are not met, there is a hand and foot reversal, the system will provide a voice reminder to the user, and the incorrect key points of the human body will be marked (e.g., marked with a red line) and displayed on the screen to remind the user.

[0079] Excessive arm bending detection: After a valid starting motion of the jumping jack is detected, the bending angle of the arm can be calculated based on the coordinates of the left and right wrists, left and right elbows, and left and right shoulder joints in the second motion video frame image. When the bending angle is greater than the preset angle, the motion is considered standard. When the bending angle is less than the preset angle, the arm is considered to be bent too much. The user is given a voice reminder, and key human body points that are incorrect are marked (e.g., connected in red) and displayed on the display interface to remind the user.

[0080] It should be noted that the steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0081] Figure 5 This is a functional block diagram of a jumping jack counting device based on key human body points provided in an embodiment of the present invention. Please refer to... Figure 5As shown, the jumping jacks counting device based on human key points is applied to image preprocessing in an optical character recognition system. It can reduce the influence of light spots and / or shadows in the image to be recognized, making the text and background in the image to be recognized more clearly separated, thus improving the accuracy of optical character recognition. The jumping jacks counting device based on human key points includes a first image acquisition module 111, a first coordinate acquisition module 112, a first action judgment unit 113, a second image acquisition module 114, a second coordinate acquisition module 115, a second action judgment unit 116, and a counting module 117.

[0082] The first image acquisition module 111 is used to monitor the video stream during the human jumping jacks exercise to acquire a first motion video frame image; the first coordinate acquisition module 112 is used to acquire the coordinates of several human key points in the first motion video frame image using a human posture detection model, as the coordinates of the first key points; the first action judgment unit 113 is used to compare the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame image according to the first key point coordinates, until the first target key point and the second target key point in a certain frame of the first motion video frame image satisfy the first positional relationship, which is recorded as a valid starting action of the jumping jacks; the second image acquisition module 114 is used to continue monitoring the human jumping jacks exercise after detecting a valid starting action of the jumping jacks. The process involves a video stream to obtain a second motion video frame image; the second coordinate acquisition module 115 is used to obtain the coordinates of several human key points in the second motion video frame image using the human posture detection model, as the second key point coordinates; the second action judgment unit 116 is used to compare the positional relationship between the first target key point in each frame of the second motion video frame image and the preset position according to the second key point coordinates, until the first target key point in a certain frame of the second motion video frame image and the preset position satisfy the second positional relationship, which is recorded as a valid end action of the jumping jack; the counting module 117 is used to increment the jumping jack count by 1, re-execute the step of monitoring the video stream during the human jumping jack movement process to obtain the first motion video frame image, until the preset stopping condition is met.

[0083] like Figure 5As shown, in this embodiment, the jumping counting device based on human body key points further includes a stable state detection module 118. The stable state detection module 118 includes a third image acquisition module, a third coordinate acquisition module, a third action judgment module, and a preset position determination module. The third image acquisition module is used to monitor the video stream during the preparation phase of the human jumping jacks to obtain preparation video frame images. The third coordinate acquisition module is used to obtain the coordinates of several key human points in the preparation video frame images using a human posture detection model. The third action judgment module is used to determine, frame by frame, whether the distance between the left and right ankles in each preparation video frame image meets the condition of feet together, wherein the condition of feet together is that the distance between the left and right ankles is less than the distance between the left and right shoulder joints. The preset position determination module is used to record the coordinates of the wrist or elbow (i.e., the first target key point) in the preparation video frame image when the distance between the left and right ankles in a certain frame of the preparation video frame image meets the condition of feet together, indicating that the human body is ready and in a stable state, and can start subsequent jumping jack movements. The module also records the coordinates of the wrist or elbow (i.e., the first target key point) in the preparation video frame image and subtracts the preset pixel value from the y-axis coordinate value of the wrist in the preparation video frame image as the coordinates of the preset position.

[0084] like Figure 5 As shown, in this embodiment, the jumping jack counting device based on human body key points further includes a motion posture correction module 119. The motion posture correction module 119 is used to detect hand and foot reversal and excessive arm bending based on the coordinates of several human body key points in the second motion video frame image, score the jumping jack action based on the detection of hand and foot reversal and excessive arm bending, and mark the erroneous human body key points with lines (e.g., the lines are marked in red), and display them through the display interface to remind the user.

[0085] It should be noted that the jumping jack counting device based on human body key points in this embodiment is a device corresponding to the jumping jack counting method based on human body key points described above. The functional modules in the jumping jack counting device based on human body key points may correspond to the corresponding steps in the jumping jack counting method based on human body key points. The jumping jack counting device based on human body key points in this embodiment can be implemented in conjunction with the jumping jack counting method based on human body key points. Accordingly, the relevant technical details mentioned in the jumping jack counting device based on human body key points in this embodiment can also be applied to the jumping jack counting method based on human body key points described above.

[0086] It should be noted that the aforementioned functional modules can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, some or all of the steps of the above method, or the aforementioned functional modules, can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0087] like Figure 6 The diagram shown is a structural schematic of an electronic device that implements the opening and closing jump counting method based on key points of the human body according to the present invention.

[0088] The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a jumping jack counting program based on key points of the human body.

[0089] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 12 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 1. Furthermore, the memory 12 can include both internal and external storage units of the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as code for counting jumping jacks based on human key points, but also to temporarily store data that has been output or will be output.

[0090] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., a jump counting program based on human body key points) and calls data stored in the memory 12 to perform various functions and process data of the electronic device 1.

[0091] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-described jumping jack counting method based on human body key points, for example... Figure 2 The steps are shown.

[0092] For example, the computer program can be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device 1. For example, the computer program can be divided into a first image acquisition module 111, a first coordinate acquisition module 112, a first action judgment unit 113, a second image acquisition module 114, a second coordinate acquisition module 115, a second action judgment unit 116, a counting module 117, a stable state detection module 118, and an action posture correction module 119.

[0093] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The software functional module, stored in the storage medium, includes several instructions to cause a computer device (which may be a personal computer, a computer device, or a network device, etc.) or processor to execute some functions of the opening and closing jump counting method based on human body key points described in various embodiments of the present invention.

[0094] The present invention relates to a jumping jack counting method, apparatus, device, and medium based on human body key points. It acquires a first motion video frame image by monitoring a video stream during a human jumping jack movement; uses a human posture detection model to obtain the coordinates of several human body key points in the first motion video frame image, which are then used as the coordinates of the first key points; based on the first key point coordinates, it compares the positional relationship between a first target key point and a second target key point in each frame of the first motion video frame image, until the first target key point and the second target key point in a certain frame of the first motion video frame image satisfy the first positional relationship, which is then recorded as a valid starting action of the jumping jack; after detecting a valid starting action of the jumping jack, it continues to monitor the human body... The video stream during the jumping jack movement is used to obtain a second motion video frame image. The coordinates of several key human points in the second motion video frame image are obtained using the human posture detection model, serving as the second key point coordinates. Based on the second key point coordinates, the positional relationship between the first target key point and a preset position in each frame of the second motion video frame image is compared frame by frame until the first target key point and the preset position in a certain frame of the second motion video frame image satisfy the second positional relationship, at which point the jumping jack movement is considered to have ended effectively. The jumping jack count is incremented by 1, and the steps of monitoring the video stream during the human jumping jack movement to obtain a first motion video frame image are repeated until a preset stopping condition is met. Using this invention, the jumping jack movement can be correctly identified, improving the accuracy of jumping jack calculation in AI-based motion detection.

[0095] The jumping jack counting method, device, equipment, and medium based on key points of the human body of the present invention can also liberate the coach's supervision to a certain extent, and clearly show the exercise score, number of repetitions, duration, calories burned, etc.

[0096] The jumping jack counting method, device, equipment, and medium based on human body key points of the present invention can also detect the reverse hand and foot posture and excessive arm bending during the jumping jack process. Based on the detection results of the reverse hand and foot posture and excessive arm bending, the jumping jack action can be scored. At the same time, the incorrect human body key points can be marked with lines (e.g., the lines are marked in red) and displayed through the display interface, thereby strengthening the user's requirements for standard posture during exercise.

[0097] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0098] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for counting jumping jacks based on key points of the human body, characterized in that, include: Monitor the video stream during the human jumping jacks exercise to obtain the first motion video frame image; The coordinates of several key human body points in the first motion video frame image are obtained using a human pose detection model and used as the coordinates of the first key point. Based on the coordinates of the first key point, the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame is compared frame by frame until the first target key point and the second target key point in a certain frame of the first motion video frame satisfy the first positional relationship, which is recorded as the effective starting action of the jumping jack. After detecting a valid starting motion of the jumping jacks, continue to monitor the video stream during the human jumping jacks motion to obtain the second motion video frame image; The coordinates of several key human body points in the second motion video frame image are obtained using the human body posture detection model, and are used as the coordinates of the second key points. Based on the coordinates of the second key point, the positional relationship between the first target key point and the preset position in each frame of the second motion video frame is compared frame by frame until the first target key point and the preset position in a certain frame of the second motion video frame satisfy the second positional relationship, which is recorded as the effective end action of the jumping jack. Increment the jumping jack count by 1, and re-execute the step of monitoring the video stream during the human jumping jack movement to obtain the first motion video frame image until the preset stopping condition is met. The first target key point includes the wrist, and the method for obtaining the preset position includes: Monitor the video stream during the preparation phase of human jumping jacks to obtain preparation video frame images; The coordinates of several key human body points in the prepared video frame image are obtained using a human pose detection model. Based on the coordinates of several key human body points in the prepared video frame image, determine frame by frame whether the distance between the left and right ankles in each prepared video frame image meets the condition of feet together. The condition of feet together is that the distance between the left and right ankles is less than the product of the distance between the left and right shoulder joints and a preset multiplier, where the preset multiplier is 0.5-1. When the distance between the left and right ankles in a certain frame of the prepared video frame meets the condition of feet together, the coordinates of the wrist in the prepared video frame are recorded, and the y-axis coordinate of the first target key point in the prepared video frame is subtracted by a preset pixel value and used as the y-axis coordinate value of the preset position.

2. The jumping jack counting method based on human body key points according to claim 1, characterized in that, The first target key point includes the wrist or elbow, and the second target key point includes the ear, eye or shoulder joint; Based on the coordinates of the first key point, the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame is compared frame by frame until the first target key point and the second target key point in a certain frame of the first motion video frame satisfy the first positional relationship. This is recorded as a valid starting action for the jumping jack, including: Based on the coordinates of the first key point, the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame is compared frame by frame until the first target key point in a certain frame of the first motion video frame is located above the second target key point, which is then recorded as a valid starting action of the jumping jack.

3. The jumping jack counting method based on human body key points according to claim 2, characterized in that, The first target key points include the left and right wrists, and the second target key points include the left and right ears; Based on the coordinates of the first key point, the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame is compared frame by frame until the first target key point in a certain frame of the first motion video frame is above the second target key point. This is then recorded as a valid starting action for the jumping jack, including: Obtain the coordinates of the left and right wrists and the left and right ears in the first motion video frame image in the current frame; Determine whether the coordinates of the left and right wrists and the coordinates of the left and right ears in the first motion video frame image in the current frame simultaneously meet the following conditions: the y-axis coordinate value of the left wrist is less than the y-axis coordinate value of the left ear, and the y-axis coordinate value of the right wrist is less than the y-axis coordinate value of the right ear. When the conditions are met, it is recorded as the effective starting action of the jumping jack; If the conditions are not met, the first motion video frame image of the next frame is used as the first motion video frame image of the current frame, and the step of obtaining the coordinates of the left and right wrists and the left and right ears in the first motion video frame image of the current frame is returned.

4. The jumping jack counting method based on human body key points according to claim 2, characterized in that, The first target key points include the left and right wrists, and the second target key points include the left and right ears; Based on the coordinates of the second key point, the positional relationship between the wrist and the preset position in each frame of the second motion video frame is compared frame by frame until the wrist and the preset position in a certain frame of the second motion video frame satisfy the second positional relationship. This is then recorded as the effective end action of the jumping jack, including: Obtain the coordinates of the left and right wrists in the second motion video frame image of the current frame; Determine whether the coordinates of the left and right wrists in the second motion video frame in the current frame and the coordinates of the preset position simultaneously meet the following conditions: the y-axis coordinate value of the left wrist is greater than the y-axis coordinate value of the preset position, and the y-axis coordinate value of the right wrist is greater than the y-axis coordinate value of the preset position. When the conditions are met, it is recorded as the effective end action of the jumping jack; If the condition is not met, the next frame of the second motion video frame is used as the current frame of the second motion video frame, and the step of obtaining the coordinates of the left and right wrists in the current frame of the second motion video frame is returned.

5. The jumping jack counting method based on human body key points according to claim 1, characterized in that, The human pose detection model is the MoveNet motion detection model.

6. The jumping jack counting method based on human body key points according to claim 1, characterized in that, The jumping jack counting method based on human body key points also includes: Based on the coordinates of the second key point, non-standard actions are detected; The jumping jacks are scored based on the results of non-standard movements, and the key human body points corresponding to the non-standard movements are marked and displayed on the interface.

7. A jumping jack counting device based on key points of the human body, characterized in that, include: The first image acquisition module is used to monitor the video stream during the human jumping jacks exercise to acquire the first motion video frame image; The first coordinate acquisition module is used to acquire the coordinates of several human key points in the first motion video frame image using a human pose detection model, and use them as the coordinates of the first key points. The first action judgment unit is used to compare the positional relationship between the first target key point and the second target key point in each frame of the first motion video frame image according to the coordinates of the first key point, until the first target key point and the second target key point in a certain frame of the first motion video frame image satisfy the first positional relationship, and then record it as a valid starting action of the jumping jack. The second image acquisition module is used to continue monitoring the video stream during the human jumping jack movement after detecting the effective starting action of the jumping jack, so as to acquire the second motion video frame image. The second coordinate acquisition module is used to acquire the coordinates of several key human body points in the second motion video frame image using the human body posture detection model, and use them as the coordinates of the second key points. The second action judgment unit is used to compare the positional relationship between the first target key point and the preset position in each frame of the second motion video frame image according to the coordinates of the second key point, until the first target key point and the preset position in a certain frame of the second motion video frame image satisfy the second positional relationship, which is recorded as the effective end action of the jumping jack. The counting module is used to increment the jumping jack count by 1 and re-execute the step of monitoring the video stream during the human jumping jack movement to obtain the first motion video frame image until the preset stopping condition is met. The first target key point includes the wrist, and the method for obtaining the preset position includes: Monitor the video stream during the preparation phase of human jumping jacks to obtain preparation video frame images; The coordinates of several key human body points in the prepared video frame image are obtained using a human pose detection model. Based on the coordinates of several key human body points in the prepared video frame image, determine frame by frame whether the distance between the left and right ankles in each prepared video frame image meets the condition of feet together. The condition of feet together is that the distance between the left and right ankles is less than the product of the distance between the left and right shoulder joints and a preset multiplier, where the preset multiplier is 0.5-1. When the distance between the left and right ankles in a certain frame of the prepared video frame meets the condition of feet together, the coordinates of the wrist in the prepared video frame are recorded, and the y-axis coordinate of the first target key point in the prepared video frame is subtracted by a preset pixel value and used as the y-axis coordinate value of the preset position.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images