Rope skipping counting method, electronic equipment and storage medium
Through video stream processing technology, the moving targets in rope skipping videos are tracked and classified, and the posture motion trajectory is generated, which solves the problems of mis-checking and high cost of rope skipping counts in the existing technology, and realizes high-accuracy automated intelligent rope skipping counting.
Patent Information
- Application Number
- CN202411917088.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-24
- Publication Date
- 2025-04-25
AI Technical Summary
The existing technology has false detection situations in the skipping rope counting, and it is impossible to accurately evaluate the jumping posture of sports personnel. The cost of intelligent skipping rope is high and depends on the rotation of the sensor for counting, so it is impossible to achieve truly automated intelligent counting.
By obtaining the rope skipping video stream, tracking each moving target, performing key point detection and pose classification, generating pose motion trajectory, and counting rope skipping based on this to avoid relying on sensor types.
It effectively avoids misdetection during rope skipping, improves the accuracy of rope skipping counting, and realizes automatic intelligent counting of rope skipping without relying on sensor types.
Smart Images
Figure CN119887839A_ABST
Abstract
Description
Technical Field
[0001] The present application generally relates to the field of image processing technology. More specifically, the present application relates to a skipping rope counting method, an electronic device and a storage medium. Background Art
[0002] With the vigorous development of sports technology, the concept of smart sports has gradually entered the public's field of vision, and the traditional fitness method of skipping rope has also ushered in a series of intelligent upgrades. The main counting methods for skipping rope are currently divided into manual counting and intelligent counting. If intelligent counting is used, high-precision sensors such as Hall sensors or infrared optical sensors need to be installed in the smart skipping rope. These sensors can detect the rotation state of the skipping rope in real time and calculate the number of skipping ropes through built-in logic. On the one hand, the cost of such smart skipping ropes is relatively high. On the other hand, it only counts the rotation of the sensor, and idling will also generate counts. It is impossible to accurately evaluate whether the athlete's skipping posture is standard and standardized, and it is difficult to achieve truly automated intelligent counting.
[0003] There are also schemes in the prior art that use videos to count rope skipping. For example, the spatiotemporal graph convolution STGCN can be used to classify and judge the input human key point topology map, so as to determine whether the human body is in the rope skipping state. After determining the rope skipping state, the trajectory of the human key points is analyzed, and the rope skipping count is performed when the upward trend is converted to a downward trend. However, this scheme cannot count single-leg, double-leg, and single- and double-leg exchange jumps separately. At the same time, since the STGCN model needs to consume a lot of computing power, the real-time detection cannot be ensured. In addition, the rope skipping action can also be judged by the Pearson correlation coefficient, and finally the number of single-leg jumps, double-leg jumps, and single- and double-leg exchange jumps is recorded through a pre-designed rope skipping counting mechanism. However, this scheme will produce more false detections for some false jumps and non-rope skipping actions, and the accuracy of rope skipping counts cannot be ensured.
[0004] In view of this, there is an urgent need to provide a rope skipping counting method so as to effectively avoid misdetection during rope skipping and improve the accuracy of rope skipping counting. Summary of the invention
[0005] In order to at least solve one or more of the technical problems mentioned above, the present application proposes a rope skipping counting method, an electronic device and a storage medium in multiple aspects. The rope skipping counting method can effectively avoid false detection during rope skipping and improve the accuracy of rope skipping counting.
[0006] In a first aspect, the present application provides a method for counting rope skipping, comprising: obtaining a rope skipping video stream; tracking each moving target in the rope skipping video stream, and performing key point detection on each moving target based on each frame of video image to obtain a key point topology map corresponding to each moving target in each frame of video image; performing posture classification on the key point topology map corresponding to each moving target in each frame of video image to obtain a valid motion posture corresponding to each moving target in each frame of video image; generating a posture motion trajectory corresponding to each moving target in each preset period based on the valid motion posture corresponding to each moving target and a preset period duration; and counting rope skipping for each moving target based on the posture motion trajectory corresponding to each moving target.
[0007] In some embodiments, tracking each moving target in the rope skipping video stream includes: extracting each frame of video image in the rope skipping video stream; performing target detection on each moving target in the video image of the current frame to obtain initial target features of each moving target; estimating the motion trajectory of each moving target in the video image of the current frame to determine the previous motion position corresponding to each moving target; adjusting the position of each moving target in the video image of the current frame based on the previous motion position corresponding to each moving target to obtain an updated video image; performing target detection on each moving target in the updated video image to obtain updated target features of each moving target; and determining whether each moving target in the rope skipping video stream is successfully tracked based on the initial target features of each moving target in the video image of the previous frame of the current frame and the updated target features of each moving target.
[0008] In some embodiments, key point detection is performed on each moving target based on each frame of video image to obtain a key point topology map corresponding to each moving target in each frame of video image, including: key point detection is performed on the human body key points of each moving target in a preset detection area in each frame of video image to obtain key point coordinate information corresponding to each human body key point of each moving target; based on the key point coordinate information corresponding to each human body key point of each moving target, a key point topology map corresponding to each moving target in each frame of video image is determined.
[0009] In some embodiments, determining a key point topology map corresponding to each moving target in each frame of video image based on key point coordinate information corresponding to each human body key point of each moving target includes: determining the key point horizontal coordinate and the key point vertical coordinate corresponding to each human body key point of each moving target as the feature vector corresponding to each human body key point of each moving target; connecting the feature vectors corresponding to each human body key point of each moving target based on the spatial connection relationship of each human body key point, so as to obtain a key point topology map corresponding to each moving target in each frame of video image.
[0010] In some embodiments, performing posture classification on the key point topology map corresponding to each moving target in each frame of video image includes: determining the adjacency matrix and node feature matrix corresponding to each frame of video image based on the key point topology map corresponding to each moving target in each frame of video image; inputting the adjacency matrix and the node feature matrix into a posture classification model to obtain a posture classification result output by the posture classification model, wherein the posture classification result includes a single-leg jumping posture, a double-leg jumping posture, and a non-rope skipping posture.
[0011] In some embodiments, determining the adjacency matrix and node feature matrix corresponding to each moving target in each frame of video image based on the key point topology map corresponding to each moving target in each frame of video image includes: obtaining the key point connection relationship corresponding to each moving target in each frame of video image in the key point topology map corresponding to each moving target in each frame of video image, and determining the adjacency matrix corresponding to each moving target in each frame of video image based on the key point connection relationship corresponding to each moving target in each frame of video image; obtaining the feature vector corresponding to each human body key point of each moving target in each frame of video image in the key point topology map corresponding to each moving target in each frame of video image, and determining the node feature matrix corresponding to each moving target in each frame of video image based on the feature vector corresponding to each human body key point of each moving target in each frame of video image.
[0012] In some embodiments, generating the posture motion trajectory corresponding to each motion target in each preset cycle based on the effective motion posture corresponding to each motion target and the preset cycle duration includes: generating the posture motion trajectory corresponding to each motion target in each preset cycle based on the displacement change of the human body center point corresponding to the effective motion posture of each motion target and the preset cycle duration.
[0013] In some embodiments, counting the number of rope jumps for each sports target based on the posture motion trajectory corresponding to each sports target includes: generating time domain vibration information corresponding to each sports target based on the posture motion trajectory corresponding to each sports target; converting the time domain vibration information corresponding to each sports target into periodic frequency domain information corresponding to each sports target; and counting the number of rope jumps for each sports target based on the periodic frequency domain information corresponding to each sports target.
[0014] In a second aspect, the present application provides an electronic device, comprising: a processor; and a memory, on which a program code for rope skipping counting is stored, and when the program code is executed by the processor, the electronic device implements the method as described above.
[0015] In a third aspect, the present application provides a non-transitory machine-readable storage medium having stored thereon a program code for rope skipping counting, and when the program code is executed by a processor, the method described above can be implemented.
[0016] The technical solution provided by this application may have the following beneficial effects:
[0017] The rope skipping counting method, electronic device and storage medium provided by the present application obtain a rope skipping video stream, and then track each moving target in the rope skipping video stream, and perform key point detection on each moving target based on each frame of video image, to obtain a key point topology map corresponding to each moving target in each frame of video image, and then perform posture classification on the key point topology map corresponding to each moving target in each frame of video image, to obtain a valid motion posture corresponding to each moving target in each frame of video image, thereby effectively avoiding misdetection of counting for non-rope skipping actions, improving counting accuracy, and at the same time being able to achieve differentiated counting for each valid motion posture.
[0018] Furthermore, the present application can generate a posture motion trajectory corresponding to each motion target in each preset period based on the effective motion posture corresponding to each motion target and the preset period duration, and then count the number of jumps for each motion target based on the posture motion trajectory corresponding to each motion target, without relying on the type of jump rope, avoiding the loss of experience caused by using a smart jump rope equipped with a sensor, and realizing automatic intelligent counting of jump ropes.
[0019] In general, the present application can effectively avoid false detection during rope skipping and improve the accuracy of rope skipping counting. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:
[0021] Figure 1 An exemplary flow chart of a method for counting rope skipping according to some embodiments of the present application is shown;
[0022] Figure 2 An exemplary flow chart showing a method for counting rope skipping according to other embodiments of the present application is shown;
[0023] Figure 3 An exemplary flow chart showing a method for counting rope skipping according to some further embodiments of the present application is shown;
[0024] Figure 4 A schematic diagram of a scene when at least one moving target is performing rope skipping in the rope skipping counting method shown in an embodiment of the present application is shown;
[0025] Figure 5A schematic diagram showing a key point topology diagram when a moving target performs single-leg rope skipping in the rope skipping counting method shown in an embodiment of the present application is shown;
[0026] Figure 6 A schematic diagram showing a key point topology diagram when a moving target performs rope skipping with both feet in a rope skipping counting method shown in an embodiment of the present application is shown;
[0027] Figure 7 A schematic diagram of the structure of an electronic device shown in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. For the simplicity and clarity of the description, the figure marks may be repeated in the drawings to indicate corresponding or similar elements when deemed appropriate. In addition, the present application sets forth many specific details in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other cases, well-known methods, processes, and components are not described in detail to avoid blurring the embodiments described herein. Moreover, the description should not be regarded as limiting the scope of the embodiments described herein. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.
[0029] It should be understood that the possible terms "first" or "second" etc. in the claims, specifications and drawings disclosed in this application are used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of this application indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.
[0030] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms. It should also be further understood that the term "and / or" used in this application specification and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.
[0031] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0032] At present, the main counting methods for skipping rope are divided into manual counting and intelligent counting. If intelligent counting is used, high-precision sensors need to be installed in the intelligent skipping rope. These sensors can detect the rotation state of the skipping rope in real time and calculate the number of skipping ropes through built-in logic. On the one hand, the cost of such intelligent skipping ropes is relatively high. On the other hand, it only counts the rotation of the sensor, and idling will also generate counts. It is impossible to accurately evaluate whether the skipping posture of the athlete is standard and standardized, and it is difficult to achieve true automatic intelligent counting. The existing solution of using video to count skipping ropes cannot count single-foot, double-foot, and single-double-foot switching jumps separately. At the same time, it requires a large number of examples and cannot ensure the real-time detection. In addition, for some fake jumps that are not skipping rope actions, there will be more false detections, and the accuracy of skipping rope counting cannot be ensured.
[0033] In view of this, there is an urgent need to provide a rope skipping counting method so as to effectively avoid misdetection during rope skipping and improve the accuracy of rope skipping counting.
[0034] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.
[0035] Figure 1 An exemplary flow chart of a method for counting rope skipping in some embodiments of the present application is shown. Figure 4 A schematic diagram of a scene when at least one moving target is performing rope skipping in the rope skipping counting method shown in an embodiment of the present application is shown; Figure 5 A schematic diagram showing a key point topology diagram when a moving target performs single-leg rope skipping in the rope skipping counting method shown in an embodiment of the present application is shown; Figure 6 A schematic diagram showing a key point topology diagram when a target performs rope skipping with both feet in a rope skipping counting method shown in an embodiment of the present application is shown. Figure 1 , Figures 4 to 6 The rope skipping counting method shown in the embodiment of the present application may include:
[0036] In step S101, a rope skipping video stream is obtained. Figure 4As shown, in an embodiment of the present application, an imaging device for collecting a rope skipping video stream can be arranged in an actual scene where a sports target is performing rope skipping, and preferably an ultra-high-definition camera can be used. The imaging field of view of the imaging device is aligned with a preset detection area, and at least one sports target can be accommodated in the preset detection area to perform rope skipping at the same time. In addition, a display screen for playing a standard rope skipping posture video can be set on the wall, which is conducive to guiding the sports target to perform accurate rope skipping. When one or more sports targets are guided on the display screen to start rope skipping, the imaging device can start collecting rope skipping video streams for rope skipping analysis and counting.
[0037] In step S102, each moving target in the rope skipping video stream is tracked, and key point detection is performed on each moving target based on each frame of video image to obtain a key point topology map corresponding to each moving target in each frame of video image. In the embodiment of the present application, key point detection refers to the detection of key points of the human skeleton. Figure 5 and Figure 6 As shown, 17 key points of the human skeleton can be detected, including 5 key points of the face and 12 key points of the trunk. The key point topology map can be determined based on the spatial connection relationship between these key points and each key point.
[0038] In step S103, the key point topology map corresponding to each moving target in each frame of video image is respectively subjected to posture classification to obtain the valid motion posture corresponding to each moving target in each frame of video image. Figure 5 and Figure 6 As shown, in the embodiment of the present application, the preset classification model can be used to Figure 5 The key point topology diagram shown is distinguished as the key point topology diagram of single-leg skipping, and its corresponding valid motion posture is single-leg skipping. Figure 6 The key point topology diagram shown is distinguished as a key point topology diagram for skipping rope with both feet, and the corresponding valid motion posture is skipping rope with both feet.
[0039] In step S104, a posture motion trajectory corresponding to each moving target in each preset period is generated based on the valid motion posture corresponding to each moving target and the preset period duration. In the embodiment of the present application, the preset period duration can be the duration obtained by evenly dividing the video stream duration of the rope skipping video stream into a number of time periods. For example, the preset period duration can be set to 1 second. In actual applications, the specific duration of the preset period duration needs to be set according to the actual application situation, and the present application does not impose any restrictions in this regard.
[0040] It can be understood that the preset cycle duration may include multiple video frames in which each moving target maintains its corresponding valid motion posture. Therefore, during the preset cycle duration, the corresponding periodic posture motion trajectory of each moving target during the current preset cycle duration can be determined through the multiple video frames corresponding to each moving target mentioned above, so that the periodic posture motion trajectories corresponding to each moving target of several preset cycle durations are spliced together, and the posture motion trajectory corresponding to each moving target in the entire video stream duration can be obtained.
[0041] In step S105, rope skipping is counted for each sport target based on the posture motion trajectory corresponding to each sport target. It can be understood that rope skipping is a sport in which the center of the human body continuously moves vertically, so a curve of the height coordinate of the center of the human body changing with time can be recorded based on the posture motion trajectory corresponding to each sport target, so as to determine the rope skipping count for each sport target according to the number of fluctuations of the curve.
[0042] The embodiment of the present application obtains the skipping video stream, and then tracks each moving target in the skipping video stream, and performs key point detection on each moving target based on each frame of video image, obtains the key point topology map corresponding to each moving target in each frame of video image, and then respectively performs posture classification on the key point topology map corresponding to each moving target in each frame of video image, and obtains the valid motion posture corresponding to each moving target in each frame of video image, so as to effectively avoid the misdetection of counting for non-skipping actions, improve the counting accuracy, and realize the distinguishing counting for each valid motion posture. Further, the present application can generate the posture motion trajectory corresponding to each moving target in each preset period based on the valid motion posture corresponding to each moving target and the preset cycle duration, and then count the skipping of each moving target based on the posture motion trajectory corresponding to each moving target, without relying on the type of skipping rope, avoiding the loss of experience generated by using the intelligent skipping rope configured with sensors, and realizing the automatic intelligent counting of skipping rope. In general, the present application can effectively avoid the misdetection during the skipping process and improve the accuracy of skipping rope counting.
[0043] In some embodiments, in order to effectively avoid misdetection of non-rope skipping movements, a posture classification model can be used to classify rope skipping postures, thereby filtering out non-rope skipping postures and determining valid exercise postures. Figure 2 Let's explain the posture classification process in detail. Figure 2 An exemplary flow chart showing a method for counting rope skipping in some other embodiments of the present application is shown in FIG. Figure 2 The rope skipping counting method shown in the embodiment of the present application may include:
[0044] In step S201, each moving target in the rope skipping video stream is tracked. In the embodiment of the present application, each frame of video image in the rope skipping video stream can be firstly extracted, and each frame of video image can be preprocessed including at least one of image denoising, image enhancement, image rotation, image scaling, etc., so as to improve image quality, reduce noise, and enhance image features, which is conducive to improving the efficiency and accuracy of subsequent analysis.
[0045] Further, ByteTrack target tracking method can be used to track each moving target in the preset detection area, and an independent ID identifier can be assigned to each moving target. Specifically, each moving target can be detected in the video image of the current frame to obtain the initial target features of each moving target. Among them, target detection can be achieved using a target detector, which can identify objects in an image or video and determine its position. It not only identifies the category of the object, but also accurately marks the position of the object by drawing a bounding box. Exemplarily, the target detector can be implemented using YOLOv8. Then, the Kalman filter algorithm can be used to estimate the motion trajectory of each moving target in the video image of the current frame, and determine the previous motion position corresponding to each moving target. Then, the position of each moving target can be adjusted in the video image of the current frame based on the previous motion position corresponding to each moving target to obtain an updated video image. After the position adjustment, target detection can be performed on each moving target in the updated video image to obtain the updated target feature of each moving target, and it can be determined whether each moving target in the rope skipping video stream is successfully tracked based on the initial target feature of each moving target in the previous frame of the video image of the current frame and the updated target feature of each moving target. Exemplarily, the Hungarian algorithm can be used to respectively match the updated target feature of each moving target with the initial target feature of each moving target in the previous frame of the video image of the current frame to determine whether the moving target in the video image of the current frame and the moving target in the previous frame of the video image of the current frame are the same target. If they are the same target, then it is determined that the tracking is successful.
[0046] In step S202, key point detection is performed on the human key points of each moving target in the preset detection area in each frame of video image to obtain key point coordinate information corresponding to each human key point of each moving target. In an embodiment of the present application, YOLOv8 can be used to perform key point detection on the human key points of each moving target in the preset detection area in each frame of video image, so as to determine the key point coordinate information corresponding to each human key point of each moving target. Among them, the key point coordinate information includes the key point horizontal coordinate and the key point vertical coordinate.
[0047] In step S203, a key point topology map corresponding to each moving target in each frame of video image is determined based on the key point coordinate information corresponding to each human key point of each moving target. In the embodiment of the present application, the key point horizontal coordinate and key point vertical coordinate corresponding to each human key point of each moving target can be first determined as the feature vector corresponding to each human key point of each moving target. Then, Figure 5 and Figure 6 As shown, based on the spatial connection relationship of each human key point, the feature vectors corresponding to each human key point of each moving target are connected respectively to obtain a key point topology map corresponding to each moving target in each frame of video image.
[0048] In step S204, the adjacency matrix and node feature matrix corresponding to each moving target in each frame of video image are determined based on the key point topology map corresponding to each moving target in each frame of video image. In an embodiment of the present application, the key point connection relationship corresponding to each moving target in each frame of video image can be obtained in the key point topology map corresponding to each moving target in each frame of video image, and the adjacency matrix corresponding to each moving target in each frame of video image is determined based on the key point connection relationship corresponding to each moving target in each frame of video image. Wherein, the aforementioned adjacency matrix is a connection relationship matrix, and the size of the adjacency matrix is determined by the number of nodes (i.e., key points) in the matrix. The elements in the matrix represent the connection relationship between the nodes.
[0049] Furthermore, the feature vector corresponding to each human key point of each moving target in each frame of video image is obtained in the key point topology map corresponding to each moving target in each frame of video image, and the node feature matrix corresponding to each moving target in each frame of video image is determined based on the feature vector corresponding to each human key point of each moving target in each frame of video image. The aforementioned node feature matrix is composed of 17 human key point feature vectors.
[0050] In step S205, the adjacency matrix and the node feature matrix are input into the posture classification model to obtain the posture classification result output by the posture classification model. In the embodiment of the present application, the posture classification result may include a single-leg hopping posture, a double-leg hopping posture, and a non-rope skipping posture.
[0051] In the embodiment of the present application, the posture classification model can adopt a graph isomorphism network (GIN, Graph Isomorphism Network). Exemplarily, in some implementation scenarios, first, image samples of single-leg hopping posture, double-leg hopping posture and non-skipping rope state can be collected and collected. Then, YOLOv8 is used to extract the human key points in the collected image samples respectively, and the horizontal coordinates and vertical coordinates of the key points are connected as the feature vectors of the points to generate a topological map, forming a human key point data sample. Then, the obtained human key point data sample can be annotated to form a three-classification topological structure data set. Then the three-classification topological structure data set is input into the initial graph isomorphism network for training, and after the training is completed, a graph isomorphism network with the ability to distinguish the skipping rope posture into a single-leg hopping posture, a double-leg hopping posture and a non-skipping rope posture can be obtained. It can be understood that in practical applications, the number of classification types of the target graph isomorphism network is diverse, specifically depending on how many types of posture image samples are collected for training, and it is necessary to determine according to the actual application situation, and the present application does not make any restrictions in this regard.
[0052] In addition, in an embodiment of the present application, the structure of the graph isomorphic network may include an input layer (input) for inputting an adjacency matrix and a node feature matrix; a hidden layer (Hidden layer), which includes a graph convolutional layer (GraphConvolutional Layer) and a graph pooling layer (Graph Pooling Layer), and the aggregation mode (Aggregate) of the graph convolutional layer and the graph pooling layer is sum, which means that when aggregating the features of neighboring nodes, the feature vectors of all neighboring nodes are added. The advantage of this is that the representation ability of the graph isomorphic network is stronger, and the posture judgment of the human body can be more accurate, effectively avoiding the counting misdetection of non-rope skipping actions; a multilayer perceptron (MLP), each graph convolution layer can include two layers of MLP; an activation function ReLU (Rectified Linear Unit, rectified linear unit), ReLU can reduce the problem of gradient disappearance due to its linear characteristics; a representation layer (Readout), which can aggregate the feature information of all nodes in the graph to obtain a feature representation of the entire graph; an output layer (Output), which is used to output the posture classification result corresponding to each moving target in each frame of video image.
[0053] Further, illustratively, the aggregation function of the aggregation method of the graph convolution layer and the graph pooling layer can be designed as follows: Let c be the feature of the key point itself, and X be the neighborhood feature set of the current key point. When X is countable, set the aggregation method Aggregate to sum, and combine (which refers to the process of merging the aggregated neighbor node features with the features of the central node and updating the feature representation of the central node) to 1+∈, there will be f(x) that makes h(c,X) injective:
[0054]
[0055] Further, any g(c,X) can be decomposed into the following f and The form satisfies the injective property:
[0056]
[0057] By introducing the multi-layer perceptron MLP, we can learn Finally, we get the GIN framework based on MLP+SUM:
[0058]
[0059] in, and f are learnable parameters. When they are 0, the current key point and the neighboring key points will be directly aggregated at the same level. The multi-layer perceptron MLP can approximate any function and thus learn the injective function, which cannot be satisfied by the single-layer perceptron.
[0060] In some embodiments, the rope skipping postures classified as non-rope skipping postures can be excluded from the valid exercise postures, and only the valid exercise postures are counted. Figure 3 Let's explain the skipping rope counting process in detail. Figure 3 An exemplary flow chart showing a method for counting rope skipping in some other embodiments of the present application is shown in FIG. Figure 3 The rope skipping counting method shown in the embodiment of the present application may include:
[0061] In step S301, the posture motion trajectory corresponding to each moving target in each preset period is generated based on the displacement change of the center point of the human body corresponding to the effective motion posture of each moving target and the preset period duration. In the embodiment of the present application, the posture motion trajectory corresponding to each moving target in each preset period can be generated based on the vertical displacement change of the center point of the human body corresponding to the effective motion posture of each moving target and the number of video frames included in the preset period duration.
[0062] In step S302, time domain vibration information corresponding to each moving target is generated based on the posture motion trajectory corresponding to each moving target. In the embodiment of the present application, the posture motion trajectory corresponding to each moving target can be mapped in a coordinate system with the horizontal axis being time and the vertical axis being height, and a vibration curve is generated in the coordinate system, thereby generating time domain vibration information corresponding to each moving target.
[0063] In step S303, the time domain vibration information corresponding to each moving target is converted into the periodic frequency domain information corresponding to each moving target. In the embodiment of the present application, in order to further improve the accuracy of rope skipping counting, the time domain vibration information corresponding to each moving target can be converted into the periodic frequency domain information corresponding to each moving target through Fourier transform, and the periodicity of the periodic frequency domain information obtained after Fourier transform is used to count rope skipping, which can effectively reduce the false detection caused by the noise in the time domain vibration information.
[0064] In step S304, rope skipping is counted for each sport target based on the periodic frequency domain information corresponding to each sport target. In the embodiment of the present application, rope skipping can be counted by the number of periodic changes in the periodic frequency domain information corresponding to each sport target, that is, one period is counted once, so that the rope skipping count can be accurately output regardless of whether the sport target is hopping on one foot or hopping on both feet.
[0065] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides an electronic device for executing a rope skipping counting method and a corresponding embodiment.
[0066] Figure 7 FIG. 7 is a block diagram showing the hardware configuration of an electronic device 700 that can implement the rope skipping counting method of an embodiment of the present application. Figure 7 As shown, the electronic device 700 may include a processor 710 and a memory 720. Figure 7 In the electronic device 700, only the components related to this embodiment are shown. Therefore, it is obvious to those skilled in the art that the electronic device 700 may also include Figure 7 The components shown in the figure are different from the common components. For example: fixed-point arithmetic units.
[0067] The electronic device 700 may correspond to a computing device having various processing functions, for example, a function for generating a neural network, training or learning a neural network, quantizing a floating-point neural network to a fixed-point neural network, or retraining a neural network. For example, the electronic device 700 may be implemented as various types of devices, such as a personal computer (PC), a server device, a mobile device, etc.
[0068] The processor 710 controls all functions of the electronic device 700. For example, the processor 710 controls all functions of the electronic device 700 by executing a program stored in the memory 720 on the electronic device 700. The processor 710 may be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an artificial intelligence processor chip (IPU), etc. provided in the electronic device 700. However, the present application is not limited thereto.
[0069] In some embodiments, the processor 710 may include an input / output (I / O) unit 711 and a computing unit 712. The I / O unit 711 may be used to receive various data, such as a skipping video stream. Exemplarily, the computing unit 712 may be used to track each moving target in the skipping video stream received via the I / O unit 711, and perform key point detection on each moving target based on each frame of video image to obtain a key point topology map corresponding to each moving target in each frame of video image; perform posture classification on the key point topology map corresponding to each moving target in each frame of video image to obtain a valid motion posture corresponding to each moving target in each frame of video image; generate a posture motion trajectory corresponding to each moving target in each preset period based on the valid motion posture corresponding to each moving target and the preset cycle duration; and count the number of skipping ropes for each moving target based on the posture motion trajectory corresponding to each moving target. The result of the skipping rope counting may be output, for example, by the I / O unit 711. The output data may be provided to the memory 720 for reading and use by other devices (not shown), or may be directly provided to other devices for use.
[0070] The memory 720 is hardware for storing various data processed in the electronic device 700. For example, the memory 720 can store processed data and data to be processed in the electronic device 700. The memory 720 can store data sets involved in the process of the rope skipping counting method that has been processed or to be processed by the processor 710, such as a rope skipping video stream, etc. In addition, the memory 720 can store applications, drivers, etc. to be driven by the electronic device 700. For example: the memory 720 can store various programs related to the rope skipping counting method to be executed by the processor 710. The memory 720 can be a DRAM, but the present application is not limited thereto. The memory 720 can include at least one of a volatile memory or a non-volatile memory. The non-volatile memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM), etc. In an embodiment, the memory 720 may include at least one of a hard disk drive (HDD), a solid state drive (SSD), a high-density flash memory (CF), a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, caches, or a memory stick.
[0071] In summary, the specific functions implemented by the memory 720 and the processor 710 of the electronic device 700 provided in the implementation mode of this specification can be explained in comparison with the aforementioned implementation modes in this specification, and can achieve the technical effects of the aforementioned implementation modes, and will not be repeated here.
[0072] In this embodiment, the processor 710 can be implemented in any suitable manner. For example, the processor 710 can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc.
[0073] It should also be understood that any module, unit, component, server, computer, terminal or device that executes instructions exemplified herein may include or otherwise access computer-readable media, such as storage media, computer storage media or data storage devices (removable and / or non-removable) such as disks, optical disks or tapes. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules or other data.
[0074] Although multiple embodiments of the present application have been shown and described herein, it is obvious to those skilled in the art that such embodiments are provided only by way of example. Those skilled in the art can think of many changes, modifications and alternatives without departing from the thought and spirit of the present application. It should be understood that in the process of practicing the present application, various alternatives to the embodiments of the present application described herein can be adopted. The attached claims are intended to limit the scope of protection of the present application, and therefore cover equivalents or alternatives within the scope of these claims.
Claims
1. A method for counting rope skipping, characterized in that: include: Get the skipping video stream; Tracking each moving target in the rope skipping video stream, and performing key point detection on each moving target based on each frame of video image, to obtain a key point topology map corresponding to each moving target in each frame of video image; Perform posture classification on the key point topology map corresponding to each moving target in each frame of video image, and obtain the effective motion posture corresponding to each moving target in each frame of video image; Generate a posture motion trajectory corresponding to each moving target in each preset period based on the effective motion posture corresponding to each moving target and the preset period duration; The rope skipping count is performed for each sports target based on the posture motion trajectory corresponding to each sports target.
2. The rope skipping counting method according to claim 1, characterized in that: The tracking of each moving target in the rope skipping video stream comprises: Extracting each frame of video image in the rope skipping video stream; Performing target detection on each moving target in the video image of the current frame to obtain initial target features of each moving target; Estimate the motion trajectory of each moving target in the video image of the current frame, and determine the previous motion position corresponding to each moving target; Adjusting the position of each moving target in the video image of the current frame based on the previous moving position corresponding to each moving target to obtain an updated video image; Performing target detection on each moving target in the updated video image to obtain an updated target feature of each moving target; Whether each moving target in the rope skipping video stream is successfully tracked is determined based on the initial target feature of each moving target in the previous frame of the current frame and the updated target feature of each moving target.
3. The rope skipping counting method according to claim 1, characterized in that: The key point detection is performed on each moving target based on each frame of video image to obtain a key point topology map corresponding to each moving target in each frame of video image, including: Perform key point detection on the human body key points of each moving target in a preset detection area in each frame of video image to obtain key point coordinate information corresponding to each human body key point of each moving target; A key point topology map corresponding to each moving target in each frame of video image is determined based on key point coordinate information corresponding to each human key point of each moving target.
4. The rope skipping counting method according to claim 3, characterized in that: Determining the key point topology map corresponding to each moving target in each frame of video image based on the key point coordinate information corresponding to each human key point of each moving target includes: Determine the key point abscissa and the key point ordinate corresponding to each human body key point of each moving target as the feature vector corresponding to each human body key point of each moving target; Based on the spatial connection relationship of each human key point, the feature vectors corresponding to each human key point of each moving target are connected respectively to obtain a key point topology map corresponding to each moving target in each frame of video image.
5. The rope skipping counting method according to claim 4, characterized in that: The posture classification of the key point topology map corresponding to each moving target in each frame of video image comprises: Determine the adjacency matrix and node feature matrix corresponding to each moving target in each frame of video image based on the key point topology map corresponding to each moving target in each frame of video image; The adjacency matrix and the node feature matrix are input into a posture classification model to obtain a posture classification result output by the posture classification model, wherein the posture classification result includes a single-leg hopping posture, a double-leg hopping posture and a non-rope skipping posture.
6. The rope skipping counting method according to claim 5, characterized in that: The determining of the adjacency matrix and the node feature matrix corresponding to each frame of video image based on the key point topology map corresponding to each moving target in each frame of video image comprises: Obtaining the key point connection relationship corresponding to each moving target in each frame of video image in the key point topology map corresponding to each moving target in each frame of video image, and determining the adjacency matrix corresponding to each moving target in each frame of video image based on the key point connection relationship corresponding to each moving target in each frame of video image; The feature vector corresponding to each human key point of each moving target in each frame of video image is obtained in the key point topology map corresponding to each moving target in each frame of video image, and the node feature matrix corresponding to each moving target in each frame of video image is determined based on the feature vector corresponding to each human key point of each moving target in each frame of video image.
7. The rope skipping counting method according to claim 1, characterized in that: The method of generating the posture motion trajectory corresponding to each moving target in each preset period based on the valid motion posture corresponding to each moving target and the preset period duration includes: The posture motion trajectory corresponding to each motion target in each preset period is generated based on the displacement change of the human body center point corresponding to the effective motion posture of each motion target and the preset period duration.
8. The rope skipping counting method according to claim 1, characterized in that: The rope skipping counting of each sports target based on the posture motion trajectory corresponding to each sports target comprises: Generate time domain vibration information corresponding to each moving target based on the posture motion trajectory corresponding to each moving target; Convert the time domain vibration information corresponding to each moving target into the periodic frequency domain information corresponding to each moving target; The rope skipping count is performed for each sports target based on the periodic frequency domain information corresponding to each sports target.
9. An electronic device, characterized in that: include: processor; as well as A memory having program codes for rope skipping counting stored therein, wherein when the program codes are executed by the processor, the electronic device implements the method as claimed in any one of claims 1 to 8.
10. A non-transitory machine-readable storage medium having stored thereon a program code for rope skipping counting, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 8 is implemented.