Rope skipping counting method and device based on action classification, medium and electronic equipment

Abstract

The invention relates to a rope skipping counting method and device based on action classification, a medium and electronic equipment, and relates to the technical field of rope skipping counting, and the method comprises the steps: obtaining an on-site rope skipping video of a target object which is a person needing rope skipping counting; based on the on-site rope skipping video, determining video key frame features and 3D pose features related to rope skipping of the target object; based on the video key frame features and the 3D pose features, determining a current rope skipping state of the target object through a preset rope skipping action classification model; if the rope skipping state is rope skipping starting, rope skipping counting is conducted on the target object; if the rope skipping state is rope skipping ending, when it is detected that the rope skipping state is rope skipping starting after the current time, recording temporary rope skipping interruption once. The rope skipping counting device has the effect of improving rope skipping counting accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of rope skipping counting, and in particular to a rope skipping counting method, device, medium and electronic equipment based on action classification. Background Art

[0002] Rope skipping is a sport game in which one or more people perform various jumping movements on a ring-shaped rope. The main materials are cotton rope, PVC (plastic) rope, bamboo rope and steel wire rope. Rope skipping counting refers to the act of counting the number of times you skip rope during the process of skipping. Rope skipping counting can help rope jumpers understand their exercise intensity and amount, so as to formulate a reasonable exercise plan and achieve better exercise results.

[0003] At present, the method commonly used for counting rope skipping is: a method based on computer vision counting, that is, detecting the key points of the rope jumper's body, and counting the rope skipping of the rope jumper based on the temporal characteristics of the key points of the body. In this method, once the rope jumper makes unnecessary movements or pseudo-rope skipping movements occur when the rope skipping is interrupted, inaccurate rope skipping counting problems such as over-counting or under-counting may occur, resulting in poor accuracy of rope skipping counting. Summary of the invention

[0004] In order to improve the accuracy of rope skipping counting, the present application provides a rope skipping counting method, device, medium and electronic device based on action classification.

[0005] In a first aspect of the present application, a method for counting rope skipping based on action classification is provided, which specifically includes: Acquire a live rope skipping video of a target object, wherein the target object is a person who needs to be counted for rope skipping; Based on the on-site rope skipping video, determining video key frame features and 3D pose features related to rope skipping of the target object; Based on the video key frame features and the 3D pose features, determining the current rope skipping state of the target object through a preset rope skipping action classification model; If the rope skipping state is the start of rope skipping, then the rope skipping count is performed for the target object; If the rope skipping state is the end of rope skipping, when the rope skipping state is detected as the start of rope skipping after the current time, a temporary interruption of rope skipping is recorded.

[0006] By adopting the above technical solution, after obtaining the live skipping video, the video key frame features and 3D pose features related to the target object's skipping are extracted from the live skipping video. The key frame details of the video key frame features can be better captured, and the 3D pose features can be used to fully reflect the target object's action posture. Further, with the video key frame features and 3D pose features, the target object's skipping action is accurately classified through the skipping action classification model, so as to more accurately determine the current skipping state of the target object. If the skipping state is to start skipping, then the target object starts to skip, so as to filter the interference of non-skipping actions on the counting to a certain extent, thereby improving the accuracy of skipping counting.

[0007] Optionally, before obtaining the on-site rope skipping video of the target object, the method further includes: Obtaining skipping video sample data corresponding to different skipping action categories, wherein the skipping action categories include starting skipping, skipping, ending skipping, and not skipping; For a single sample of the skipping rope video, human body detection is performed using a preset target detection algorithm, and after the human body detection is completed, pose detection is performed using a preset pose detection algorithm to obtain corresponding pose sequence data; Based on the pose sequence data, determining a corresponding 3D pose feature sample, and for a single rope skipping video sample data, determining a corresponding rope skipping action video key frame; Based on the key frame of the rope skipping action video, the corresponding key frame feature sample is determined, and the 3D pose feature sample and the key frame feature sample corresponding to the same rope skipping video sample data are fused to obtain a fused feature; Based on the fused features corresponding to each of the rope skipping video sample data, a preset untrained action classification model is trained to obtain a rope skipping action classification model.

[0008] By adopting the above technical solution, after obtaining the skipping rope video sample data of different skipping rope action categories, the corresponding 3D pose feature samples and key frame feature samples are extracted from each skipping rope video sample data, and the features of the two dimensions of 3D pose feature samples and key frame feature samples are integrated to train the model, so that the training effect of the model is better and the performance of the trained model is better, thereby enabling the skipping rope action classification model to more accurately identify and classify the skipping rope actions of the target object.

[0009] Optionally, counting the number of rope skipping of the target object specifically includes: Based on the on-site rope skipping video, the rope skipping of the target object is counted in real time to obtain the number of rope skipping and detect whether there is a rope; If so, when the rope skipping state is detected to be rope skipping end after the current time, the number of rope skipping in the interval from the current time to the rope skipping end is determined as the number of rope skipping; If not, when the rope skipping state is detected to be rope skipping end after the current time, the number of rope skipping in the interval from the current time to the rope skipping end is determined as the number of ropeless rope skipping.

[0010] By adopting the above technical scheme, during the process of rope skipping counting, if the presence of a rope is detected during the rope skipping process, it means that the target object has been performing rope skipping before the end of the rope skipping, and the number of rope skipping counted in real time will be determined as the number of rope skipping with rope; otherwise, the number of rope skipping counted in real time will be determined as the number of rope skipping without rope, thereby achieving accurate counting of the rope skipping of the target object.

[0011] Optionally, based on the on-site rope skipping video, counting the rope skipping of the target object in real time to obtain the number of rope skipping specifically includes: Based on the on-site rope skipping video, determining at least one human body core key point corresponding to the target object; Performing filtering processing on all the core key points of the human body to obtain filtered key points; The number of up and down reciprocating movements of the filtering key point is counted to obtain the number of rope skipping corresponding to the target object.

[0012] By adopting the above technical solution, the core key points of the human body are filtered, thereby removing the noise in the core key points of the human body and improving the data quality. Finally, since a single up-and-down reciprocating motion reflects a single skipping process in which the target object jumps up and then falls from a high place, the number of up-and-down reciprocating motions of the filtering key points is determined as the number of skipping ropes, thereby more accurately counting the skipping ropes of the target object.

[0013] Optionally, the detecting whether there is a rope specifically includes: When it is detected that the filtering key point has an up-and-down reciprocating motion, the period of time during which the single up-and-down reciprocating motion occurs in the live rope skipping video is determined as a counting period; Determine whether there is a first video frame and a second video frame in the rope skipping video segment corresponding to the counting period, the first video frame being a video frame in which a rope is detected at an upper half of the target object, and the second video frame being a video frame in which a rope is detected at a lower half of the target object; If so, it is determined that there is a rope, if not, it is determined that there is no rope.

[0014] By adopting the above technical scheme, a single up and down reciprocating motion indicates that the target object performs a single rope skipping exercise. If the rope appears in the upper and lower parts of the target object respectively during the counting period corresponding to the single rope skipping exercise, it means that there is a rope during the target object's rope skipping process. Otherwise, it means that there is no rope during the rope skipping process, thereby more accurately judging whether there is a rope during the target object's rope skipping process.

[0015] Optionally, the method further includes: When it is currently determined that the target object has temporarily interrupted rope skipping, obtaining at least one actual rope skipping height and a corresponding actual wind speed of the target object from the beginning of rope skipping to the current time, wherein the actual wind speed is the wind speed when the rope skipping height of the target object is the actual rope skipping height; Obtain the wind speed intervals in which the historical wind speeds were located when the rope jumpers temporarily stopped rope jumping, count the first occurrence times of each wind speed interval, and select a first number of wind speed intervals from each wind speed interval in descending order of the first occurrence times to determine as key wind speed intervals; Obtaining the height intervals of historical rope skipping heights at which historical rope jumpers temporarily interrupted rope skipping when the historical wind speed was in a single key wind speed interval, counting the second number of occurrences of each of the height intervals, and selecting a second number of height intervals from each of the height intervals in descending order of the second number of occurrences, and determining them as key height intervals corresponding to the single key wind speed interval; Determine a first weight for each of the key wind speed intervals, and determine a second weight for the key height interval corresponding to each of the key wind speed intervals, wherein the first weight is a ratio of a first occurrence number of each key wind speed interval to a sum of first occurrence numbers of all key wind speed intervals, and the second weight is a ratio of a second occurrence number of a single key height interval corresponding to the key wind speed interval to a sum of second occurrence numbers of all corresponding key height intervals; Based on the first weight, the second weight, the actual rope skipping height and the corresponding actual wind speed, a temporary interruption of rope skipping of the current target object is verified.

[0016] By adopting the above technical solution, the greater the number of first occurrences, the more likely the rope jumper will have a temporary interruption in rope jumping when the wind speed is in the corresponding wind speed interval, thereby determining the key wind speed interval; the greater the number of second occurrences, the more likely the rope jumper will have a temporary interruption in rope jumping when the rope jumper's rope jumping height is in the corresponding height interval in a single key wind speed interval, thereby determining the key height interval. Finally, combining the first weight and the second weight, the possibility of a temporary interruption in rope jumping is analyzed, thereby realizing the verification of the temporary interruption of rope jumping of the target object, thereby making the record of temporary interruption of rope jumping more accurate.

[0017] Optionally, the checking of whether the current target object has temporarily interrupted rope skipping based on the first weight, the second weight, the actual rope skipping height and the corresponding actual wind speed specifically includes: The key height interval where the single actual rope skipping height is located is determined as the target height interval, and if there is a corresponding actual wind speed in the key wind speed interval, the corresponding key wind speed interval is determined as the target wind speed interval; If the target height interval exists in each key height interval corresponding to the target wind speed interval, the corresponding target wind speed interval is determined as an important wind speed interval, and the product of the first weight of each important wind speed interval and the second weight of the corresponding target height interval is calculated; When the current temporary interruption of rope skipping is the first interruption from the start of rope skipping to the current process, the products are summed to obtain the sum of the products. If the sum of the products exceeds a preset threshold, it is verified that the current target object has a temporary interruption of rope skipping.

[0018] By adopting the above technical solution, the product of the first weight of each important wind speed interval and the second weight of the corresponding target height interval is calculated. The larger the product, the greater the possibility of temporary interruption of rope skipping when the actual wind speed is in the important wind speed interval and the actual rope skipping height of the target object is in the corresponding target height interval. Finally, when the current temporary interruption of rope skipping is the first interruption from the start of rope skipping to the current process, the sum of the products is summed to obtain the sum of the products. If the sum of the products is greater than the preset threshold, it means that the overall possibility of temporary interruption of rope skipping is relatively high in the target object from the start of rope skipping to the current time interval, and there is a high probability of temporary interruption of rope skipping at present, then it is verified that the current target object does have a temporary interruption of rope skipping.

[0019] In a second aspect of the present application, a rope skipping counting device based on action classification is provided, which specifically comprises: An information acquisition module, used to acquire a live rope skipping video of a target object, wherein the target object is a person who needs to be counted for rope skipping; A feature determination module, used to determine the video key frame features and 3D pose features related to the target object's rope skipping based on the on-site rope skipping video; An action classification module, used to determine the current rope skipping state of the target object through a preset rope skipping action classification model based on the video key frame features and the 3D pose features; A rope skipping counting module, configured to count the number of rope skippings of the target object if the rope skipping state is the start of rope skipping; The interruption recording module is used to record a temporary interruption of rope skipping when the rope skipping state is detected to be the start of rope skipping after the current time if the rope skipping state is the end of rope skipping.

[0020] By adopting the above technical solution, the information acquisition module obtains the on-site rope skipping video of the target object, the feature determination module determines the video key frames and 3D pose features related to the rope skipping of the target object, and then the action classification module determines the current rope skipping state of the target object based on the video key frame features and 3D pose features through a preset rope skipping action classification model, and the rope skipping counting module counts the rope skipping of the target object when the rope skipping state is the start of rope skipping. Finally, when the rope skipping state is the end of rope skipping, the interruption recording module records a temporary rope skipping interruption when it detects that the rope skipping state is the start of rope skipping after the current time.

[0021] In a third aspect of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method steps described in any one of the first aspects are performed.

[0022] In a fourth aspect of the present application, an electronic device is provided, specifically comprising: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the processor is used to load and execute the computer program stored in the memory so that the electronic device performs the method as described in any one of the first aspects.

[0023] In summary, the present application includes at least one of the following beneficial technical effects: After obtaining the live skipping video, the video key frame features and 3D pose features related to the target object's skipping are extracted from the live skipping video. The video key frame features can better capture the key frame details of the skipping action, and the 3D pose features can fully reflect the target object's action posture. Further, with these video key frame features and 3D pose features, the skipping action of the target object is accurately classified through the skipping action classification model, so as to more accurately determine the current skipping state of the target object. If the skipping state is to start skipping, then the skipping count of the target object is started, so as to filter the interference of non-skipping actions on the count to a certain extent, thereby improving the accuracy of skipping count. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of a method for counting rope skipping based on action classification provided in an embodiment of the present application; Figure 2 It is a structural schematic diagram of a rope skipping counting device based on action classification provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of another rope skipping counting device based on action classification provided in an embodiment of the present application.

[0025] Explanation of the accompanying drawings: 11. Information acquisition module; 12. Feature determination module; 13. Action classification module; 14. Rope skipping counting module; 15. Interruption recording module; 16. Model training module; 17. Record verification module. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0027] In the description of the embodiments of the present application, words such as "exemplarily", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily", "for example" or "for example" is intended to present related concepts in a concrete way.

[0028] In the description of the embodiments of the present application, the term "and / or" is only a kind of association relationship describing the associated objects, indicating that there may be three kinds of relationships, for example, A and / or B, which can represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0029] See also Figure 1 The present application embodiment discloses a flow chart of a method for counting rope skipping based on action classification, which can be implemented by a computer program or run on a rope skipping counting device based on action classification based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application, specifically including: S101: Acquire a live rope skipping video of a target object.

[0030] Specifically, the target object is a person who needs to count rope skipping. The on-site rope skipping video is a video that records the target object's rope skipping in an outdoor environment in real time. In other embodiments, the current action video can also be a video that records the target object's rope skipping in an indoor environment in real time. In addition, the execution subject of the rope skipping counting method based on action classification disclosed in the embodiment of the present application is a server, and the server communicates wirelessly with the terminal and the camera. The terminal can be a personal computer or a smart phone. A client or applet related to rope skipping counting is installed in the terminal, and the server can be a background server of the client or applet, specifically an independent physical server, or a server cluster composed of multiple physical servers.

[0031] Furthermore, an implementation scenario of the rope skipping counting method based on action classification disclosed in the present application is: rope skipping is counted for students during rope skipping training in school physical education classes, or rope skipping is counted for examinees in physical education examinations. When rope skipping counts are needed, after the rope jumper is ready, the personnel sends a command to start counting to the server through the client in the terminal, and the server obtains the command, obtains the live video of the area where the target object is rope skipping in real time through the camera, and obtains the live rope skipping video, which only contains the target object. Different rope skipping areas are provided with corresponding cameras. In other embodiments, the live rope skipping video of the target object obtained by the camera may include other rope skipping personnel in addition to the target object.

[0032] S102: Based on the on-site rope skipping video, determine the video key frame features and 3D pose features related to the rope skipping of the target object.

[0033] Specifically, a video keyframe refers to a frame with a specific meaning and function in a video. In an embodiment of the present application, a video keyframe is a picture frame of a key action reflecting the state of a target object skipping rope in a live rope skipping video. Exemplarily, a video keyframe in a live rope skipping video can be a frame of a picture when the target object jumps to the highest point in the air in the video, or a frame of a picture at the moment when the target object takes off in the video. Video keyframe features refer to the characteristics and properties possessed by video keyframes in a live rope skipping video. After obtaining the live rope skipping video, the image data of the video keyframes related to the target object skipping rope is obtained from the live rope skipping video through a preset Final Cut Pro X tool. Then, video keyframe features are extracted from the image data through a yolo series feature skeleton structure, such as yolov5 or yolov7.

[0034] Furthermore, the 3D pose feature refers to the characteristic information describing the posture of the target object when skipping rope in three-dimensional space. The 3D pose feature can capture the specific spatial position of each part of the body of the target object when skipping rope, including the coordinates of the arms, legs, head and other parts in three-dimensional space, so as to present the skipping rope state of the target object more comprehensively and accurately. In an embodiment of the present application, a feasible way to determine the 3D pose feature is: the human body of the target object in the live skipping rope video is detected by a preset target detection algorithm, the target detection algorithm adopts the yolo series detection algorithm, and then the pose of the target object in the live skipping rope video is detected by a preset pose detection algorithm to obtain the pose data corresponding to this live skipping rope video. Finally, the heat map features are extracted from the pose data through the 3D convolution structure to obtain the 3D pose feature.

[0035] S103: Based on the video key frame features and the 3D pose features, the current rope skipping state of the target object is determined through a preset rope skipping action classification model.

[0036] Specifically, after the video key frame features and 3D pose features are determined, the video key frame features and 3D pose features are input into a preset rope skipping action classification model, the rope skipping action of the target object in the live rope skipping video is identified and classified, and the current rope skipping state of the target object is finally determined, so as to facilitate more accurate rope skipping counting of the target object. The rope skipping state includes but is not limited to starting rope skipping, rope skipping, rope skipping end and rope skipping not being performed. Starting rope skipping indicates the movement process of the target object entering rope skipping from never jumping, that is, the movement process of entering the state of shaking the rope with both hands and jumping from the state of never starting to jump. Rope skipping indicates the movement process of the target object jumping with both feet or jumping with one or two feet alternately. Rope skipping ends indicate the movement process of the target object from jumping to stopping rope skipping, mainly referring to the movement process from shaking the rope with both hands and continuously jumping to stopping shaking the rope or stopping jumping. Not jumping indicates that the target object is not in the action of rope skipping. Exemplarily, if the rope skipping action classification model determines that the current rope skipping action classification of the target object is starting rope skipping, then the current rope skipping state of the target object is starting rope skipping. It should be noted that the rope skipping action classification model is a trained convolutional neural network model. In other embodiments, the rope skipping action classification model may also be a trained recurrent neural network model.

[0037] Furthermore, before obtaining the live skipping video of the target object, there is a process of determining the skipping action classification model, and the specific process is as follows: collect the historical skipping videos of the students recorded in the camera history, the historical skipping videos contain the video of the entire process of a single person skipping, and then use the video editing tool to edit the skipping videos containing only a single skipping action category from each historical skipping video, that is, the skipping video sample data. The skipping action categories include starting skipping, skipping, ending skipping, and not skipping. Among them, the video editing tool can be Adobe Premiere Pro tool, and in other embodiments, it can also be Final Cut Pro X tool.

[0038] Further, for a single skipping rope video sample data, a human target in the video is detected by a preset target detection algorithm, that is, the existence position and approximate area of ​​the human body are located in the skipping rope video sample data. The target detection algorithm can be Faster R-CNN based on deep learning, and in other embodiments, it can also be a yolo series detection algorithm. After the human body detection is completed, pose detection is performed on the skipping rope video sample data by a preset pose detection algorithm, and finally the pose sequence data corresponding to the skipping rope video sample data is obtained, that is, a series of continuous human posture information in the skipping rope video sample data, including the posture changes of the human body over a period of time. Among them, the pose detection algorithm can be an hrnet detection algorithm, and in other embodiments, it can also be an OpenPose algorithm.

[0039] Further, according to the pose sequence data, the corresponding 3D pose feature sample is determined. In the embodiment of the present application, a feasible determination method is: converting the pose sequence data into json file data, constructing the heat map data under the time series based on the json file data through Gaussian mapping, and finally extracting the heat map features from the heat map data through the 3D convolution structure to obtain the 3D pose feature sample corresponding to the rope skipping video sample data. Then, for a single rope skipping video sample data, determine the corresponding rope skipping action video key frame, that is, determine the video key frame in the rope skipping video sample data. Then, based on the rope skipping action video key frame, extract the corresponding key frame feature sample. The specific determination method can be found in step S102, which will not be repeated here.

[0040] Furthermore, the 3D pose feature samples and key frame feature samples corresponding to the same skipping rope video sample data are fused to obtain fused features. This reduces the number of channels that the model needs to process, improves the quality of the sample data input to the model, reduces the computational cost, and improves the performance of the model. Specifically, feature fusion can be performed through Channel Fusion. In other embodiments, feature fusion can also be performed through principal component analysis. Among them, Channel Fusion is a technology widely used in the fields of image processing and computer vision, which refers to the process of fusing image data from multiple sources or multiple channels.

[0041] Finally, the fused features corresponding to each skipping video sample data are input into a preset untrained action classification model for training. During the training process, supervision is constrained by a cross entropy damage function until the model converges, and a skipping action classification model that can identify the category of a person's skipping action based on 3D pose features and key frame features is obtained. Among them, the untrained action classification model is a convolutional neural network model. This process is a prior art and will not be repeated here. It should be noted that compared to the conventional skipping action classification based only on 3D pose data during the skipping process, the present application also simultaneously integrates the video key frames during the skipping process, which can more timely capture the moment when a person starts or ends skipping, making the accuracy of the skipping action classification higher.

[0042] S104: If the rope skipping state is to start rope skipping, the rope skipping count is performed for the target object.

[0043] Specifically, if the current rope skipping action category of the target object is determined to be rope skipping start through the rope skipping action classification model, and the rope skipping state is determined to be rope skipping start, then rope skipping counts of the target object are started. A feasible implementation method is: through the preset OpenPose algorithm, based on the on-site rope skipping video, the various human body key points of the target object are detected, including but not limited to the head, neck, shoulders, elbows, wrists, hips, knees and ankles, etc. Then at least one human body core key point is selected from each human body key point, and the human body core key point refers to an important position point that plays a key role in describing the action posture of the target object rope skipping. In an embodiment of the present application, the human body core key points include key points in the feet and hands of the target object. Further, each human body core key point is filtered to obtain a filtered key point. Specifically, a mean filter is performed through a one-euro filter (OneEuroFillter) to remove the noise in the human body core key point and improve the data quality.

[0044] Finally, the number of up and down reciprocating movements of the filter key points is counted. Specifically, the OpenPose algorithm can be used to observe the vertical movement pattern of the filter key points according to the position change trajectory of the filter key points in the continuous frames of the live skipping video. If the filter key points continue to make periodic position changes in the up and down direction for a period of time, it is determined that there is up and down reciprocating movement, and the number of up and down reciprocating movements is counted to obtain the number of skipping ropes corresponding to the target object.

[0045] While counting the number of rope skipping of the target object, it is also necessary to detect whether there is a rope based on the on-site rope skipping video. If the rope is detected during the rope skipping process of the target object, it means that the rope skipping performed by the target object is with a rope. Then, when the rope skipping state is detected to be the end of rope skipping after the current time, the number of rope skipping within the interval from the current time to the end of rope skipping is determined as the number of rope skipping; conversely, if the rope is not detected, it means that the rope skipping performed by the target object is without a rope. Then, when the rope skipping state is detected to be the end of rope skipping after the current time, the number of rope skipping within the interval from the current time to the end of rope skipping is determined as the number of ropeless rope skipping. Among them, a feasible detection method for detecting whether there is a rope is as follows: when the filtering key point is detected to have up and down reciprocating motion, the time period of the single up and down reciprocating motion in the live rope skipping video is determined as a counting period, or a counting cycle, and then rope detection is performed on each video frame of the rope skipping video segment corresponding to the counting period in the live rope skipping video. If there is a rope in the upper part of the target object in the video frame, then it is determined that there is a first video frame in the rope skipping video segment; if there is a rope in the lower part of the target object in the video frame, then it is determined that there is a second video frame in the rope skipping video segment. When the first video frame and the second video frame exist in the rope skipping video segment at the same time, it is determined that there is a rope in the rope skipping process of the target object; otherwise, it is determined that there is no rope in the rope skipping process of the target object.

[0046] S105: If the rope skipping state is rope skipping end, when the rope skipping state is detected as rope skipping start after the current time, a temporary rope skipping interruption is recorded.

[0047] Specifically, if the skipping rope status is the end of skipping rope, it means that the target object may have temporarily interrupted skipping rope because the rope hit the feet, or the target object's skipping rope task may be completed, then continue to classify the target object's real-time skipping rope movements, if it is determined again that the target object has a skipping rope action category of starting skipping rope, that is, the skipping rope status is detected as starting skipping rope after the current time, indicating that the target object is temporarily interrupted in skipping rope, then it is recorded as a temporary interruption of skipping rope.

[0048] In other embodiments, when it is currently determined that the target object has a temporary interruption in rope skipping, it means that the target object may have made a mistake in rope skipping, such as the rope hitting the feet, resulting in a temporary interruption of rope skipping, rather than the end of rope skipping. Then, at least one actual rope skipping height and the corresponding actual wind speed of the target object from the beginning of rope skipping to the current time are obtained. Specifically, the actual wind speed is obtained through a preset wind speed sensor, and the distance between the target object's feet and the ground is measured through a preset distance sensor, thereby obtaining the actual rope skipping height. Among them, the actual wind speed is the wind speed when the target object's rope skipping height is the actual rope skipping height. Exemplarily, the actual rope skipping height A and the corresponding actual wind speed B can be understood as when the target object's rope skipping height is A, the ambient wind speed is B.

[0049] Furthermore, based on the historical records of rope skipping interruptions, the wind speed intervals in which the historical wind speeds were located when the rope skipping of historical rope skippers was temporarily interrupted are obtained, and the first number of occurrences of each wind speed interval is counted. The larger the first number of occurrences, the more likely the rope skipper is to have a temporary interruption of rope skipping when the wind speed is in the corresponding wind speed interval. In descending order of the first number of occurrences, the first number of wind speed intervals are selected from each wind speed interval to be determined as key wind speed intervals, i.e., wind speed intervals where rope skipping is prone to temporary interruption. The historical records include, but are not limited to, the rope skipping height and corresponding wind speed of historical rope skippers who had temporary interruptions of rope skipping. Historical rope skippers are rope skippers with the same height, weight, physical health status, and other information as the target object.

[0050] Furthermore, based on the above historical records, the height intervals of historical rope skipping heights where rope skipping was temporarily interrupted by historical rope jumpers when the historical wind speed was in a single key wind speed interval were obtained, and the second number of occurrences of each height interval was counted. The larger the second number of occurrences, the more likely it is that rope skipping will be temporarily interrupted when the rope jumper's rope skipping height is in the corresponding height interval under a single key wind speed interval. Then, in descending order of the second number of occurrences, a second number of height intervals are selected from each height interval to be determined as the key height interval corresponding to the key wind speed interval, that is, the height interval where rope skipping is prone to temporary interruption. It should be noted that the wind direction when the historical rope jumper was jumping rope is consistent with the wind direction when the target object was jumping rope, and both are the wind directions that have the greatest impact on rope skipping.

[0051] Further, a first weight for each key wind speed interval is determined, and a second weight for the key height interval corresponding to each key wind speed interval is determined, wherein the first weight is the ratio of the first occurrence number of each key wind speed interval to the sum of the first occurrence numbers of all key wind speed intervals, and the second weight is the ratio of the second occurrence number of a single key height interval corresponding to the key wind speed interval to the sum of the second occurrence numbers of all corresponding key height intervals.

[0052] Finally, based on the first weight, the second weight, the actual rope skipping height and the corresponding actual wind speed, the temporary interruption of rope skipping of the current target object is verified, so that the result of the temporary interruption of rope skipping determined in step S105 is more accurate. A feasible implementation method is: the key height interval where the single actual rope skipping height is located is determined as the target height interval. If the actual wind speed corresponding to the single actual rope skipping height exists in the key wind speed interval, then the corresponding key wind speed interval is determined as the target wind speed interval. Further, if the above-mentioned target height interval exists in each key height interval corresponding to the target wind speed interval, then the corresponding target wind speed interval is determined as the important wind speed interval. Next, the product of the first weight of each important wind speed interval and the second weight of the corresponding target height interval is calculated. The larger the product, when the actual wind speed is in the important wind speed interval, when the actual rope skipping height of the target object is in the corresponding target height interval, the greater the possibility of a temporary interruption of rope skipping. Finally, when the current temporary interruption of rope skipping is the first interruption from the start of rope skipping to the current process, the products are summed up to obtain the sum of the products. If the sum of the products is greater than the preset threshold, it means that the overall possibility of a temporary interruption of rope skipping for the target object from the start of rope skipping to the current time interval is relatively high, and there is a high probability of a temporary interruption of rope skipping at present. In this way, it is verified that the current target object does have a temporary interruption of rope skipping, and at the same time, the accuracy of the rope skipping action classification model in action classification can be verified.

[0053] In one embodiment, when the current temporary interruption of rope skipping is not the first interruption from the start of rope skipping to the current process, the temporary interruption of rope skipping from the start of rope skipping to the current process is determined as a historical interruption, and the product exceeding a preset product threshold is selected from various products to be determined as the product to be analyzed, and if the actual wind speed and the actual rope skipping height corresponding to a single historical interruption are both in the important wind speed range and the target height range corresponding to the product to be analyzed, then it is determined that the record of the historical interruption is correct.

[0054] In one embodiment, if the target object is counted for skipping rope during physical exercise, the current skipping rope height and the corresponding current wind speed of the target object are obtained, and the key wind speed interval in which the current wind speed is located is determined as the reference wind speed interval. If the current skipping rope height exists in the key height interval corresponding to this reference wind speed interval, the corresponding key height interval is determined as the reference height interval, and the first weight product of the first weight of this reference wind speed interval and the second weight of the corresponding reference height interval is calculated. If the first weight product is greater than a preset product threshold, it means that the target object continues to skip rope at the current skipping rope height, and there is a high possibility of temporary interruption of skipping rope. Then, the second weight product of the first weight of the reference wind speed interval and the second weight of each corresponding key height interval is calculated, and the minimum second weight product is selected from each second weight product. When the minimum second weight product is less than the product threshold, the key height interval corresponding to the minimum second weight product is determined as the recommended height interval. Finally, a skipping rope height adjustment reminder is sent to the target object for this recommended height interval, thereby reducing the risk of temporary interruption of skipping rope, ensuring the continuity of skipping rope exercise to a certain extent, and improving the training effect.

[0055] In another embodiment, if the rope skipping area is indoors in summer, then the coverage area of ​​a single indoor air conditioner is obtained, and the rope skipping heights of multiple rope jumpers in the coverage area are obtained, and the key wind speed interval where the wind speed of the single air conditioner is located is determined as the wind speed interval that needs attention. If the key height interval corresponding to the wind speed interval that needs attention includes the rope skipping height, then the corresponding key height interval is determined as the height interval that needs attention. Calculate the third weight product of the first weight of the wind speed interval that needs attention and the second weight of each corresponding height interval that needs attention, and sum each third weight product to obtain the sum result. If the sum result is greater than the preset threshold, then issue a reminder to reduce the air conditioner wind speed, thereby reducing the impact on the rope skipping of the rope jumper.

[0056] The implementation principle of the rope skipping counting method based on action classification in the embodiment of the present application is as follows: after obtaining the live rope skipping video, extract the video key frame features and 3D pose features related to the rope skipping of the target object from the live rope skipping video. Through the video key frame features, the key frame detail information recording the rope skipping action can be better captured, and through the 3D pose features, the action posture of the target object can be fully reflected. Further, with this video key frame feature and 3D pose feature, the rope skipping action of the target object is accurately classified through the rope skipping action classification model, so as to more accurately determine the current rope skipping state of the target object. If the rope skipping state is to start rope skipping, then start rope skipping counting for the target object, so as to filter the interference of non-rope skipping actions on the counting to a certain extent, thereby improving the accuracy of rope skipping counting.

[0057] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0058] See also Figure 2 , which is a schematic diagram of the structure of a rope skipping counting device based on action classification provided by an embodiment of the present application. The rope skipping counting device based on action classification can be implemented as all or part of the device through software, hardware or a combination of both. The device includes an information acquisition module 11, a feature determination module 12, an action classification module 13, a rope skipping counting module 14 and an interruption recording module 15.

[0059] The information acquisition module 11 is used to acquire a live rope skipping video of a target object, where the target object is a person who needs to be counted for rope skipping; A feature determination module 12 is used to determine video key frame features and 3D pose features related to the target object skipping rope based on the on-site skipping rope video; An action classification module 13 is used to determine the current rope skipping state of the target object through a preset rope skipping action classification model based on the video key frame features and the 3D pose features; A skipping counting module 14 is used to count the skipping of the target object if the skipping state is to start skipping; The interruption recording module 15 is used to record a temporary interruption of rope skipping if the rope skipping state is the end of rope skipping and when it is detected that the rope skipping state is the start of rope skipping after the current time.

[0060] Optional, such as Figure 3 As shown, the device also includes a model training module 16, which is specifically used to: Obtaining skipping video sample data corresponding to different skipping action categories, where the skipping action categories include starting skipping, skipping, ending skipping, and not skipping; For a single skipping rope video sample data, human body detection is performed using a preset target detection algorithm. After human body detection is completed, pose detection is performed using a preset pose detection algorithm to obtain corresponding pose sequence data. Based on the pose sequence data, the corresponding 3D pose feature samples are determined, and for a single rope skipping video sample data, the corresponding rope skipping action video key frames are determined; Based on the key frames of the rope skipping action video, the corresponding key frame feature samples are determined, and the 3D pose feature samples and key frame feature samples corresponding to the same rope skipping video sample data are fused to obtain the fused features; Based on the fused features corresponding to each rope skipping video sample data, the preset untrained action classification model is trained to obtain a rope skipping action classification model.

[0061] Optionally, the rope skipping counting module 14 is specifically used for: Based on the on-site rope skipping video, the target object's rope skipping is counted in real time to obtain the number of rope skipping and detect whether there is a rope; If so, when the rope skipping state is detected to be rope skipping end after the current time, the number of rope skipping in the interval from the current time to the rope skipping end is determined as the number of rope skipping; If not, when the rope skipping state is detected to be rope skipping end after the current time, the number of rope skipping in the interval from the current time to the rope skipping end is determined as the number of ropeless rope skipping.

[0062] Optionally, the rope skipping counting module 14 is specifically used for: Based on the on-site rope skipping video, determine at least one human body core key point corresponding to the target object; Filter all the core key points of the human body to obtain filtered key points; The number of up and down reciprocating movements of the filtering key points is counted to obtain the number of rope skipping corresponding to the target object.

[0063] Optionally, the rope skipping counting module 14 is specifically used for: When it is detected that the filtering key point has an up-and-down reciprocating motion, the period of time during which the single up-and-down reciprocating motion occurs in the on-site rope skipping video is determined as the counting period; Determine whether there is a first video frame and a second video frame in the rope skipping video segment corresponding to the counting period, the first video frame being a video frame in which the rope is detected at the upper half of the target object, and the second video frame being a video frame in which the rope is detected at the lower half of the target object; If so, it is determined that there is a rope, if not, it is determined that there is no rope.

[0064] Optionally, the device further includes a record verification module 17, which is specifically used to: When it is determined that the target object temporarily stops skipping rope, obtaining at least one actual skipping rope height and corresponding actual wind speed of the target object from the beginning of skipping rope to the current time, where the actual wind speed is the wind speed when the skipping rope height of the target object is the actual skipping rope height; Obtain the wind speed intervals in which the historical wind speeds were located when the rope jumpers temporarily stopped rope jumping, count the first occurrence times of each wind speed interval, and select the first number of wind speed intervals from each wind speed interval in descending order of the first occurrence times to determine as key wind speed intervals; Obtain the height interval of the historical rope skipping height at which the historical rope jumper temporarily interrupted rope skipping when the historical wind speed was in a single key wind speed interval, count the second occurrence number of each height interval, select a second number of height intervals from each height interval in descending order of the second occurrence number, and determine them as the key height intervals corresponding to the single key wind speed interval; Determine a first weight for each key wind speed interval, and determine a second weight for the key height interval corresponding to each key wind speed interval, the first weight being the ratio of the first number of occurrences of each key wind speed interval to the sum of the first number of occurrences of all key wind speed intervals, and the second weight being the ratio of the second number of occurrences of a single key height interval corresponding to the key wind speed interval to the sum of the second number of occurrences of all corresponding key height intervals; Based on the first weight, the second weight, the actual rope skipping height and the corresponding actual wind speed, a temporary interruption of rope skipping of the current target object is verified.

[0065] Optionally, the record verification module 17 is specifically used for: The key height interval where a single actual rope skipping height is located is determined as the target height interval. If there is a corresponding actual wind speed in the key wind speed interval, the corresponding key wind speed interval is determined as the target wind speed interval. If there is a target height interval in each key height interval corresponding to the target wind speed interval, the corresponding target wind speed interval is determined as an important wind speed interval, and the product of the first weight of each important wind speed interval and the second weight of the corresponding target height interval is calculated; When the current temporary interruption of rope skipping is the first interruption from the start of rope skipping to the current process, the products are summed to obtain the sum of the products. If the sum of the products exceeds a preset threshold, it is verified that the current target object has a temporary interruption of rope skipping.

[0066] It should be noted that the rope skipping counting device based on action classification provided in the above embodiment only uses the division of the above functional modules as an example when executing the rope skipping counting method based on action classification. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the rope skipping counting device based on action classification provided in the above embodiment and the rope skipping counting method embodiment based on action classification belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0067] An embodiment of the present application further discloses a computer-readable storage medium, and the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, a rope skipping counting method based on action classification of the above embodiment is adopted.

[0068] Among them, the computer program can be stored in a computer-readable medium, the computer program includes computer program code, the computer program code can be in the form of source code, object code, executable file or certain middleware, etc. The computer-readable medium includes any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer-readable medium includes but is not limited to the above-mentioned components.

[0069] Among them, through this computer-readable storage medium, a skipping rope counting method based on action classification of the above embodiment is stored in a computer-readable storage medium, and is loaded and executed on a processor to facilitate the storage and application of the above method.

[0070] An embodiment of the present application further discloses an electronic device, in which a computer program is stored in a computer-readable storage medium. When the computer program is loaded and executed by a processor, the above-mentioned rope skipping counting method based on action classification is adopted.

[0071] The electronic device may be a desktop computer, a laptop computer, a cloud server or other electronic device, and the electronic device includes but is not limited to a processor and a memory. For example, the electronic device may also include input and output devices, a network access device, and a bus.

[0072] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0073] Among them, the memory can be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device, or it can be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD) or a flash memory card (FC) equipped on the electronic device. Moreover, the memory can also be a combination of an internal storage unit and an external storage device of the electronic device. The memory is used to store computer programs and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or is to be output, and this application does not impose any restrictions on this.

[0074] Among them, through this electronic device, a skipping rope counting method based on action classification of the above embodiment is stored in the memory of the electronic device, and is loaded and executed on the processor of the electronic device for easy use.

[0075] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. This application is intended to cover any variation, use or adaptive change of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A rope skipping counting method based on action classification, characterized in that: The method comprises: Acquire a live rope skipping video of a target object, wherein the target object is a person who needs to be counted for rope skipping; Based on the on-site rope skipping video, determining video key frame features and 3D pose features related to the rope skipping of the target object; Based on the video key frame features and the 3D pose features, determining the current rope skipping state of the target object through a preset rope skipping action classification model; If the rope skipping state is the start of rope skipping, then the rope skipping count is performed for the target object; If the rope skipping state is the end of rope skipping, when the rope skipping state is detected as the start of rope skipping after the current time, a temporary interruption of rope skipping is recorded.

2. The rope skipping counting method based on action classification according to claim 1 is characterized in that: Before obtaining the on-site rope skipping video of the target object, the method further includes: Obtaining skipping video sample data corresponding to different skipping action categories, wherein the skipping action categories include starting skipping, skipping, ending skipping, and not skipping; For a single sample of the skipping rope video, human body detection is performed using a preset target detection algorithm, and after the human body detection is completed, pose detection is performed using a preset pose detection algorithm to obtain corresponding pose sequence data; Based on the pose sequence data, determining a corresponding 3D pose feature sample, and for a single rope skipping video sample data, determining a corresponding rope skipping action video key frame; Based on the key frame of the rope skipping action video, the corresponding key frame feature sample is determined, and the 3D pose feature sample and the key frame feature sample corresponding to the same rope skipping video sample data are fused to obtain a fused feature; Based on the fused features corresponding to each of the rope skipping video sample data, a preset untrained action classification model is trained to obtain a rope skipping action classification model.

3. The rope skipping counting method based on action classification according to claim 1, characterized in that: The rope skipping counting of the target object specifically includes: Based on the on-site rope skipping video, the rope skipping of the target object is counted in real time to obtain the number of rope skipping and detect whether there is a rope; If so, when the rope skipping state is detected to be rope skipping end after the current time, the number of rope skipping in the interval from the current time to the rope skipping end is determined as the number of rope skipping; If not, when the rope skipping state is detected to be rope skipping end after the current time, the number of rope skipping in the interval from the current time to the rope skipping end is determined as the number of ropeless rope skipping.

4. The rope skipping counting method based on action classification according to claim 3 is characterized in that: The step of counting the number of rope skipping of the target object in real time based on the on-site rope skipping video to obtain the number of rope skipping specifically includes: Based on the on-site rope skipping video, determining at least one human body core key point corresponding to the target object; Performing filtering processing on all the core key points of the human body to obtain filtered key points; The number of up and down reciprocating movements of the filtering key point is counted to obtain the number of rope skipping corresponding to the target object.

5. The rope skipping counting method based on action classification according to claim 4 is characterized in that: The detecting whether there is a rope specifically includes: When it is detected that the filtering key point has an up-and-down reciprocating motion, the occurrence period of the single up-and-down reciprocating motion in the live rope skipping video is determined as the counting period; Determine whether there is a first video frame and a second video frame in the rope skipping video segment corresponding to the counting period, the first video frame being a video frame in which a rope is detected at an upper half of the target object, and the second video frame being a video frame in which a rope is detected at a lower half of the target object; If so, it is determined that there is a rope, if not, it is determined that there is no rope.

6. The rope skipping counting method based on action classification according to claim 1 is characterized in that: The method further comprises: When it is currently determined that the target object has temporarily interrupted rope skipping, obtaining at least one actual rope skipping height and a corresponding actual wind speed of the target object from the beginning of rope skipping to the current time, wherein the actual wind speed is the wind speed when the rope skipping height of the target object is the actual rope skipping height; Obtain the wind speed intervals in which the historical wind speeds were located when the rope jumpers temporarily stopped rope jumping, count the first occurrence times of each wind speed interval, and select a first number of wind speed intervals from each wind speed interval in descending order of the first occurrence times to determine as key wind speed intervals; Obtaining the height intervals of historical rope skipping heights at which historical rope jumpers temporarily interrupted rope skipping when the historical wind speed was in a single key wind speed interval, counting the second number of occurrences of each of the height intervals, and selecting a second number of height intervals from each of the height intervals in descending order of the second number of occurrences, and determining them as key height intervals corresponding to the single key wind speed interval; Determine a first weight for each of the key wind speed intervals, and determine a second weight for the key height interval corresponding to each of the key wind speed intervals, wherein the first weight is a ratio of a first occurrence number of each key wind speed interval to a sum of first occurrence numbers of all key wind speed intervals, and the second weight is a ratio of a second occurrence number of a single key height interval corresponding to the key wind speed interval to a sum of second occurrence numbers of all corresponding key height intervals; Based on the first weight, the second weight, the actual rope skipping height and the corresponding actual wind speed, a temporary interruption of rope skipping of the current target object is verified.

7. The rope skipping counting method based on action classification according to claim 6 is characterized in that: The checking of whether the current target object has temporarily interrupted rope skipping based on the first weight, the second weight, the actual rope skipping height and the corresponding actual wind speed specifically includes: The key height interval where the single actual rope skipping height is located is determined as the target height interval, and if there is a corresponding actual wind speed in the key wind speed interval, the corresponding key wind speed interval is determined as the target wind speed interval; If the target height interval exists in each key height interval corresponding to the target wind speed interval, the corresponding target wind speed interval is determined as an important wind speed interval, and the product of the first weight of each important wind speed interval and the second weight of the corresponding target height interval is calculated; When the current temporary interruption of rope skipping is the first interruption from the start of rope skipping to the current process, the products are summed to obtain the sum of the products. If the sum of the products exceeds a preset threshold, it is verified that the current target object has a temporary interruption of rope skipping.

8. A rope skipping counting device based on action classification, characterized in that: include: An information acquisition module (11) is used to acquire a live rope skipping video of a target object, wherein the target object is a person whose rope skipping needs to be counted; A feature determination module (12) is used to determine the video key frame features and 3D pose features related to the target object's rope skipping based on the on-site rope skipping video; An action classification module (13) is used to determine the current rope skipping state of the target object through a preset rope skipping action classification model based on the video key frame features and the 3D pose features; A rope skipping counting module (14), configured to count the number of rope skippings of the target object if the rope skipping state is the start of rope skipping; The interruption recording module (15) is used for recording a temporary interruption of rope skipping when the rope skipping state is detected to be the start of rope skipping after the current time if the rope skipping state is the end of rope skipping.

9. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is loaded and executed by a processor, the method according to any one of claims 1 to 7 is adopted.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor loads and executes the computer program, the method according to any one of claims 1 to 7 is adopted.

Images