A method for constructing a skipping rope recognition network model based on space-time graph convolution segmentation and application thereof

Abstract

The application discloses a construction method of a skipping rope recognition network model based on space-time graph convolution segmentation and application thereof, which comprises the following steps: collecting video data samples; performing 3D human key point detection on the video data samples to obtain sample human key point time series; performing frame segmentation and labeling on the sample human key point time series, and taking the result as a label of training data; establishing a space-time graph convolution neural network; training the space-time graph convolution neural network by using the sample human key point time series as training data; defining a loss function, updating parameters through back propagation, and obtaining a skipping rope recognition network model; and the above technical scheme adopts the space-time graph convolution neural network training, defines the loss function, and updates the space-time graph convolution neural network parameters through the back propagation, so that the space-time graph convolution neural network can accurately predict the label of the training data, the network model gradually learns the correct skipping rope action classification and recognition, and the accuracy of the model is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent motion detection technology, and in particular to a method for constructing a rope skipping recognition network model based on spatiotemporal graph convolution segmentation and its application. Background Technology

[0002] With rope skipping being included in the middle school entrance examination, the number of students participating in rope skipping in primary and secondary schools across the country is increasing. However, due to the fast-paced nature of rope skipping and the complexity of events such as tripping over the rope, counting rope skips requires a large amount of manpower, and the accuracy of the counting cannot be guaranteed. With the continuous development of artificial intelligence and deep learning technologies, automatic rope skipping counting and statistics can be performed using intelligent algorithms and equipment, which can reduce the large amount of manpower required for physical education examinations and improve the accuracy of the test results.

[0003] In this context, the action recognition method based on spatiotemporal graph convolution is applied to automatic rope skipping counting and recognition. It mainly uses target detection to obtain the human body position, then uses human body key point detection technology to obtain the key points of the human body, and judges the up and down movement process of the human body in the vertical plane based on the local point or overall motion state of the key points, and then judges the count based on the obtained sine curve. See the patent application publication number CN 115346149 A, which discloses a rope skipping counting method and system based on spatiotemporal graph convolutional networks. This method includes: acquiring image data; acquiring the coordinates of key human body points; generating a topological map of key human body points; determining the human body state; analyzing the human body movement trend (when the human body is in a rope skipping state, the trajectory of the key human body points is smoothed and analyzed to determine the human body movement trend); rope skipping counting, which counts rope skipping based on the human body movement trend, and counts rope skipping when the human body movement trend changes from an upward trend to a downward trend. This invention utilizes a spatiotemporal graph convolutional neural network to analyze the results of human posture estimation, obtain the rope skipping state of the person skipping the rope, and perform accurate rope skipping counting. It has high counting accuracy, can automatically identify the start and end of rope skipping, and will not cause miscounting due to other actions. Although the above-mentioned technical solutions mainly focus on the recognition of action types, they can only determine whether the current user is jumping rope for a certain period of time, but it is difficult to detect abnormal states during rope jumping, such as preparatory actions before rope jumping, tripping while jumping rope, key point displacement caused by the back and forth movement of the human body, and incorrect counting caused by various styles of rope jumping. Summary of the Invention

[0004] Therefore, a jump rope recognition method based on spatiotemporal graph convolution segmentation is needed to solve the above problems.

[0005] To achieve the above objectives, this invention provides a method for constructing a rope skipping recognition network model based on spatiotemporal graph convolution segmentation, which includes the following steps:

[0006] Collect video data samples;

[0007] Perform 3D human keypoint detection on video data samples to obtain the time series S of human keypoints in the samples;

[0008] The time series of key points of the human body sample S is segmented into frames and labeled, and the result is used as the label of the training data, denoted as L.

[0009] A spatiotemporal graph convolutional neural network is established, which includes a backbone part and a segmentation head part. The backbone part is used for spatiotemporal feature extraction, and the segmentation head part performs frame segmentation for spatiotemporal feature extraction.

[0010] The spatiotemporal graph convolutional neural network is trained using the time series of key points of human samples S as training data; a loss function is defined, and the parameters of the spatiotemporal graph convolutional neural network are updated through backpropagation to obtain the rope skipping recognition network model.

[0011] Furthermore, the step of collecting video data samples includes the following steps:

[0012] n are set up in front of the rope skipping position. c One auxiliary camera;

[0013] n c An auxiliary camera captures the rope skipping motion, and n timestamps are obtained through synchronization. c Video data samples from an auxiliary camera.

[0014] Furthermore, the step of performing frame segmentation and labeling on the time series of key points of the human body sample S, and using the result as the label of the training data, denoted as L, includes the following steps:

[0015] A calibrated camera is placed directly in front of the rope skipping position;

[0016] The calibration camera collects calibration video data, which is collected synchronously with the video data sample video data;

[0017] Perform 3D human key point detection on the calibration video data to obtain the time series of the calibration human key points;

[0018] Perform curve manipulation on the human key points in the time series of calibrated human key points to obtain motion characteristic curves;

[0019] The motion feature curve is used to find local maxima and minima as intervals for frame segmentation, thereby obtaining several segmentation intervals for calibrating the time series of human key points, and these segmentation intervals are then labeled.

[0020] Based on the synchronization timestamp and several segmentation intervals and annotations of the time series of human body key points, the time series S of human body key points of the sample is segmented into frames and labeled, and the result is used as the label of the training data and denoted as L.

[0021] Furthermore, the labeling includes one of the following states: jumping, falling, tripping rope interruption, and other abnormal states.

[0022] Furthermore, the backbone part includes an STGC module, which is used for spatiotemporal feature extraction and outputting a feature map; the segmentation head part includes a Permute component and an Average Pool2D component, the Permute component performs dimensionality transformation on the feature map output by the backbone part, and the Average Pool2D component performs two-dimensional average pooling on the output of the Permute component.

[0023] Furthermore, the backbone part also includes an Upsample component and a Concat component. The Upsample component performs bilinear interpolation on the output feature map, upsampling it in the time and space dimensions to obtain a high-resolution feature map. The Concat component stacks high-resolution feature maps of different scales in the channel dimension to perform multi-scale information fusion. The segmentation head part also includes filtering the average pooled feature map using the NMC method and taking the class with the highest confidence as the segmentation result output.

[0024] Furthermore, the loss function is a segment coupling loss function. and cross-entropy loss function Integration, then

[0025]

[0026]

[0027]

[0028] Among them, S end This represents the actual end frame of each sub-action segment. S represents the predicted end frame for each sub-action segment. strat Represents the actual start frame of each sub-action segment. L represents the start frame for prediction of each sub-action segment; i The i-th element of the label L representing the training data. The representative model predicts the label. The i-th element.

[0029] A rope skipping recognition method based on spatiotemporal graph convolution segmentation, comprising the following steps, utilizes the rope skipping recognition network model generated by the aforementioned method for constructing a rope skipping recognition network model based on spatiotemporal graph convolution segmentation:

[0030] Acquire video data;

[0031] Perform 3D human keypoint detection on video data to obtain the time series of human keypoints;

[0032] The time series of human body key points are input into the rope skipping recognition network model for frame segmentation and labeling to obtain the rope skipping status;

[0033] The number of jump rope states is accumulated for each state, including the jump state, the fall state, the tripping and rope interruption state, and other abnormal states.

[0034] A storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for rope skipping recognition based on spatiotemporal graph convolution segmentation.

[0035] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the above-described method for rope skipping recognition based on spatiotemporal graph convolution segmentation.

[0036] Unlike existing technologies, the above technical solution uses the backbone of a spatiotemporal graph convolutional neural network to extract spatiotemporal features, capturing the spatial and temporal changes in rope skipping movements and providing a more comprehensive feature representation, which helps to better distinguish different rope skipping movements. The segmentation head part is used to perform frame segmentation of spatiotemporal features, dividing the rope skipping movements into segments in time and identifying the start and end times of each movement, thereby achieving more granular rope skipping movement recognition. By defining a loss function as the error metric for calculating the label L between the output of the spatiotemporal graph convolutional neural network and the training data, the parameters of the spatiotemporal graph convolutional neural network are updated through backpropagation, enabling it to accurately predict the labels of the training data. This allows the network model to gradually learn the correct rope skipping movement classification and recognition, improving the model's accuracy. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating a method for constructing a rope skipping recognition network model based on spatiotemporal graph convolution segmentation according to the present invention.

[0038] Figure 2 This is a schematic diagram of the camera deployment structure in a method for constructing a rope skipping recognition network model based on spatiotemporal graph convolution segmentation according to the present invention.

[0039] Figure 3This is a schematic diagram of the spatiotemporal graph convolutional neural network structure of the present invention;

[0040] Figure 4 This is a schematic diagram of the spatiotemporal graph convolutional neural network structure of the present invention.

[0041] Explanation of reference numerals in the attached figures:

[0042] 10. Auxiliary camera;

[0043] 20. Calibrate the camera. Detailed Implementation

[0044] To explain in detail the technical content, structural features, objectives, and effects of the technical solution, the following description is provided in conjunction with specific embodiments and accompanying drawings.

[0045] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0046] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0047] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0048] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0049] Unless otherwise specified, the use of terms such as “comprising,” “including,” “having,” or other similar expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0050] Similar to the interpretation in the Patent Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also interpreted in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0051] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0052] Unless otherwise expressly specified or limited, the terms "installation," "connection," "linking," "fixing," and "setting," as used in the description of the embodiments of this application, should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two components or the interaction between two components. For those skilled in the art to which this application pertains, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0053] See Figures 1-4 As shown, this invention provides a method for constructing a rope skipping recognition network model based on spatiotemporal graph convolution segmentation, which includes the following steps:

[0054] S1. Collect video data samples;

[0055] S2. Perform 3D human key point detection on the video data sample to obtain the time series S of the human key point of the sample.

[0056] S3. Perform frame segmentation and labeling on the time series of key points of the human body sample S, and use the result as the label of the training data, denoted as L.

[0057] S4. Establish a Spatial Temporal Graph Convolutional Network (ST-GCN), which includes a backbone part and a segmentation head part. The backbone part is used for spatiotemporal feature extraction, and the segmentation head part performs frame segmentation for spatiotemporal feature extraction.

[0058] S5. Use the time series of key points of the human body sample S as training data to train the spatiotemporal graph convolutional neural network; define the loss function, update the parameters of the spatiotemporal graph convolutional neural network through backpropagation, and obtain the rope skipping recognition network model.

[0059] The aforementioned video data samples are video data from the start and end of a rope skipping exercise. The start of the rope skipping exercise includes the preparation phase, and the end includes the exit action. The aforementioned human keypoints can be key points where the human posture changes significantly during the rope skipping exercise. They can be used to monitor and evaluate one or more of the following: hip joint center, right hip joint, right knee joint, right ankle, left hip joint, left knee joint, left ankle, pelvis, chest, neck, head, right shoulder joint, right elbow joint, right wrist, left shoulder joint, left elbow joint, and left wrist. The aforementioned 3D human keypoint detection of the video data samples to obtain the sample human keypoint time series S refers to importing the video data samples into a 3D human keypoint detection algorithm for processing, obtaining the coordinates of the sample human keypoints and their corresponding times, and normalizing them according to the time dimension to obtain the sample human keypoint time series S. Of course, preprocessing can be performed before importing the video data samples into the 3D human keypoint detection model, such as video frame extraction and resolution adjustment. The aforementioned 3D human keypoint detection algorithm can be OpenPose, HRNet, MMPose, etc. The preferred human 3D keypoint detection algorithm is MMPose. Based on the open-source nature of MMPose, it can be modified and customized according to needs to meet the pose estimation requirements in different scenarios.

[0060] By performing 3D human keypoint detection on video data samples, keypoints of the human body in the sample can be obtained in the time series. These keypoints provide more detailed human posture information, which helps to identify rope skipping movements. The time series of keypoints is then segmented and labeled, and the rope skipping state is extracted from the keypoint time series as labels for training data. This allows the spatiotemporal graph convolutional neural network to learn the temporal information of the rope skipping movement, resulting in better classification and recognition. The backbone of the spatiotemporal graph convolutional neural network is used for spatiotemporal feature extraction, capturing the spatial and temporal aspects of the rope skipping movement. The changes provide a more comprehensive feature representation, which helps to better distinguish different rope skipping movements; the segmentation of the Head part is used to perform frame segmentation of spatiotemporal features, dividing the rope skipping movements into segments in time, identifying the start and end time points of each movement, thereby achieving more granular rope skipping movement recognition; by defining a loss function as the error metric for calculating the label L between the output of the spatiotemporal graph convolutional neural network and the training data label, the parameters of the spatiotemporal graph convolutional neural network are updated through backpropagation, enabling it to accurately predict the label of the training data, allowing the network model to gradually learn the correct rope skipping movement classification and recognition, and improving the accuracy of the model.

[0061] The step S1 above, which involves collecting video data samples, may include the following steps:

[0062] n are set up in front of the rope skipping position. c One auxiliary camera;

[0063] n c Ten auxiliary cameras capture the rope skipping motion, and the n data are obtained through synchronized timestamps. c Video data samples from 10 auxiliary cameras.

[0064] Then, by synchronizing the timestamp, n can be obtained. c Multi-view, time-synchronized video data from 10 auxiliary cameras during the start and end of a rope skipping exercise, assuming the data duration is f frames. Then, the time series of key points on the human body is obtained. Where R represents the dimension of the video data sample, 3 represents the three dimensions (x, y, z) of the keypoint, g represents the number of human keypoints, and f represents the duration of the video data sample; the labels of the training data are... That is, when the key points of the human body are the center of the hip joint, the right hip joint, the right knee joint, the right ankle, the left hip joint, the left knee joint, the left ankle, the pelvis, the chest, the neck, the head, the right shoulder joint, the right elbow joint, the right wrist, the left shoulder joint, the left elbow joint, and the left wrist,

[0065] See Figure 2 As shown, the above n cThe positions of the 10 auxiliary cameras can be arbitrarily placed along the positive xyz axes centered on the ground at the jump rope's location. This can be achieved by deploying n... c Each auxiliary camera 10 can observe rope skipping from multiple angles and perspectives, increasing the richness of video data samples and more comprehensively capturing and analyzing rope skipping movements, such as preparatory movements before rope skipping, rope tripping, key point offsets caused by the forward and backward movement of the human body, and various styles of rope skipping, providing a more accurate data foundation and improving the generalization ability of the trained model.

[0066] In step S2 above, the step of segmenting and labeling the time series of human body key points S in the sample, and using the result as the label L for training data, is performed manually in some embodiments, and through certain specific steps in others. This invention provides a step comprising the following steps:

[0067] A calibration camera 20 is installed directly in front of the rope skipping position;

[0068] The calibration camera 20 collects calibration video data, which is collected synchronously with the video data sample video data.

[0069] Perform 3D human key point detection on the calibration video data to obtain the time series of the calibration human key points;

[0070] By performing curve manipulation on the human key points in the time series of calibrated human key points, motion characteristic curves can be obtained;

[0071] The motion feature curve is used to find local maxima and minima as intervals for frame segmentation, thereby obtaining several segmentation intervals for calibrating the time series of human key points, and these segmentation intervals are then labeled.

[0072] Based on the synchronization timestamp and several segmentation intervals and annotations of the time series of human body key points, the time series S of human body key points of the sample is segmented into frames and labeled, and the result is used as the label of the training data and denoted as L.

[0073] The aforementioned calibration camera 20 is positioned directly in front of the rope-jumping position to capture the complete rope-jumping trajectory. For example, it could be positioned 250cm in front of the rope-jumping position and 160cm above the ground. The aforementioned 3D human keypoint detection of the calibration video data to obtain the calibration human keypoint time series refers to importing the calibration video data into a 3D human keypoint detection algorithm for processing, obtaining the coordinates of the calibrated human keypoints and their corresponding times, and normalizing them along the time dimension to obtain the calibration human keypoint time series S. Of course, preprocessing can be performed before importing the calibration video data into the 3D human keypoint detection model, such as video frame extraction and resolution adjustment. The aforementioned 3D human keypoint detection algorithm can be OpenPose, HRNet, MMPose, etc. MMPose is the preferred 3D human keypoint detection algorithm because its open-source nature allows for modification and customization to meet pose estimation requirements in different scenarios. The above-mentioned curve manipulation of human key points in the time series of calibrated human key points to obtain motion characteristic curves refers to finding the average value of any one or more key points and observing its vertical height direction in the time series (see...). Figure 2 The changes in the z-axis (of the curve) form a motion characteristic curve. Preferably, curve smoothing is performed at key points on the human body, i.e., jitter removal. Jitter removal can be performed using methods such as exponentially weighted shift adjustment or average shift adjustment. By removing jitter, the curve becomes smoother, improving data readability, accuracy, and analytical effectiveness, making the curve smoother and easier to understand. The aforementioned window is a preset parameter that determines the accuracy and sensitivity of detecting local extrema on the curve. Smaller windows can more easily detect subtle changes in the curve, while larger windows can detect a wider range of extrema. Therefore, choosing an appropriate window size is crucial. The window can be set to 3s, 4s, 5s, etc., specifically depending on the speed of the jump rope movement. The above annotations include one of the following states: jumping state, falling state, rope interruption state, and other abnormal states.

[0074] By calibrating cameras 20 and n c Ten auxiliary cameras simultaneously collect data, enabling n cThe video data samples collected by the auxiliary camera 10 are identical in time series to the calibration video data collected by the calibration camera 20. That is, the frame segmentation and annotation of the sample human keypoint time series are the same as those of the calibration human keypoint time series. Therefore, only frame segmentation and annotation of the calibration human keypoint time series are needed to obtain the frame segmentation and annotation of the sample human keypoint time series. Furthermore, by performing curve manipulation on the human keypoints in the calibration human keypoint time series, motion feature curves can be obtained. The local maxima and minima in the motion feature curves can be used as the basis for frame segmentation. Local maxima and minima typically represent the peaks and troughs of motion, corresponding to key actions or turning points in the video or motion sequence. Using local maxima and minima as the start and end points of frames, continuous frames are divided, and the video or action sequence is divided into different time periods. This allows for better capture of key actions. By using the height range of local maxima and minima, states are determined, such as jumping, falling, rope tripping, and other abnormal states. Other abnormal states can include rope preparation at the start of rope skipping and leaving at the end of rope skipping. By labeling the determined states, several segmentation intervals and labels of the human body key point time series can be obtained. Then, the sample human body key point time series S is segmented and labeled according to the synchronization timestamp, and the result is used as the label of the training data, denoted as L, which is the label of the spatiotemporal graph convolutional neural network training data.

[0075] In step S4, the backbone part includes the STGC module, which is used for spatiotemporal feature extraction and output feature map; the segmentation head part includes the Permute component and the Average Pool2D component, the Permute component performs dimensional transformation on the feature map output by the backbone part, and the Average Pool2D component performs two-dimensional average pooling on the output of the Permute component.

[0076] See Figure 3 As shown, this invention provides an implementation method where STGC block (STGC module) represents STGC block (input channel, output channel), that is, the input channel size and output channel size of the STGC module; Permute(0,3,1,2) represents the dimensional transformation of the high-dimensional tensor, bringing the frame number dimension to the second dimension, which is convenient for average pooling to retain frame information; Average Pool2D(k=(3,17)) represents using a pooling kernel of size (3,17) for two-dimensional average pooling.

[0077] See Figure 4To improve the accuracy of spatiotemporal graph convolutional neural networks (SPCNNs) in tasks involving high-fine-grained feature extraction and representation, the SPCNN is further improved by including an Upsample and a Concat component in the backbone. The Upsample component performs bilinear interpolation on the feature map output by the STGC module, upsampling it in both temporal and spatial dimensions to obtain a high-resolution feature map. The Concat component stacks high-resolution feature maps of different scales along the channel dimension, performing multi-scale information fusion and outputting the result. The segmentation head also includes filtering the average pooled feature map using the NMC method, selecting the class with the highest confidence as the segmentation result.

[0078] By using the Upsample component to build a multi-scale pyramid structure to fuse temporal and spatial features, rich motion features in both the temporal and spatial dimensions during human motion can be extracted from the hidden layer. Due to the inclusion of multi-scale information, the backbone output of the model yields three feature maps. Non-maximum confidence (NMC) filtering is proposed, and the class with the highest confidence in the motion state of each frame is used as the segmentation result.

[0079] In step S5 above, the loss function is a function that measures the difference between the model's predicted result and the true label. By defining the loss function, the output of the spatiotemporal graph convolutional neural network is made as close as possible to the true label, helping the model to train better feature representations and parameter weights, thereby improving the model's performance and generalization ability. The loss function can be a segment coupling loss function. and cross-entropy loss function Integration, then

[0080]

[0081]

[0082]

[0083] Wherein, S in equation (2) end This represents the actual end frame of each sub-action segment. S represents the predicted end frame for each sub-action segment. strat Represents the actual start frame of each sub-action segment. L represents the start frame for prediction of each sub-action segment; i The i-th element of the label L representing the training data. The representative model predicts the label. The i-th element.

[0084] Segmentation Coupling Loss is a loss function used in image segmentation tasks to measure the difference between the segmentation result and the true label. Its main feature is combining pixel-level classification loss with the spatial continuity of the segmentation result. Cross-Entropy Loss is a loss function commonly used in classification tasks, measuring the difference between the predicted result and the true label. It can effectively drive the model to classify into the correct category. Its main feature is that by minimizing the cross-entropy loss function, it can help the model learn a more accurate classification boundary and class probability distribution, thereby improving the model's classification performance. Combining these two loss functions to form a custom loss function leverages their respective advantages. The segmentation coupling loss function can solve the class imbalance problem, ensuring that the model learns more accurately on positive samples, while the cross-entropy loss function can drive the model to classify into the correct category. By combining them, the class distribution and classification accuracy of the training data can be comprehensively considered, thereby improving the model's performance and generalization ability.

[0085] This invention also provides a rope skipping recognition method based on spatiotemporal graph convolution segmentation. The rope skipping recognition network model generated by the above-mentioned method for constructing a rope skipping recognition network model based on spatiotemporal graph convolution segmentation includes the following steps:

[0086] Acquire video data;

[0087] Perform 3D human keypoint detection on video data to obtain the time series of human keypoints;

[0088] The time series of human body key points is input into the rope skipping recognition network module for frame segmentation and labeling to obtain the rope skipping status;

[0089] The number of jump rope states is accumulated for each state, including the jump state, the fall state, the tripping and rope interruption state, and other abnormal states.

[0090] By using a rope skipping recognition network model trained with a spatiotemporal graph convolutional neural network, the temporal sequence features and dynamic changes of rope skipping movements can be better captured, thereby improving the accuracy of rope skipping state recognition. By accumulating each type of annotation, the frequency or duration of each movement state can be statistically analyzed, resulting in more comprehensive rope skipping recognition results.

[0091] This invention also provides a storage medium storing a computer program that, when executed by a processor, implements the steps of the jump rope recognition method based on spatiotemporal graph convolution segmentation described above. The computer program involved in the embodiments can be stored in a computer device readable storage medium, including but not limited to disks, magnetic tapes, magnetic cards, floppy disks, flash memory, optical disks, optical cards, read-only memory (ROM), random access memory (RAM), erasable programmable ROM (EPROM), and electrically erasable programmable ROM (EEPROM), etc., as well as other biological, physical, or chemical structures capable of performing similar or equivalent functions to the storage media listed above, such as DNA, RNA, proteins, and other units with information storage capabilities. In specific embodiments, the storage medium can be one of the above-mentioned media types or a combination of the above-mentioned media types. In different embodiments, the computer program involved in the embodiments can be centrally stored in a single medium or distributed across multiple media. The memory containing the computer device readable storage medium can be non-volatile memory or random access memory. These computer device readable storage media can be built into the device or connected to the device of the embodiments as an external device or part of an external device. In some embodiments, the memory having the computer device readable storage media is deployed locally; in other embodiments, the memory can be deployed remotely from the processor, for example, as a network-attached memory accessed via RF circuitry or an external port and a communication network, wherein the communication network can be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), or a suitable combination thereof, as long as it enables computer device access to the memory. Furthermore, the computer programs involved in the embodiments can be stored in plaintext / ciphertext form or designed as training data, which can be integrated and recombined through model training and implicitly stored in the parameter states of deep neural networks or other machine learning models.

[0092] This invention also provides an electronic device including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the steps of the above-described method for rope skipping recognition based on spatiotemporal graph convolution segmentation. The processor described in this application embodiment can be implemented by hardware, firmware, software, or a combination thereof. It can be a circuit, one or more of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, or a microprocessor. It also includes other physical, biological, or chemical structures capable of implementing functions similar to or equivalent to the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., thereby enabling the processor to execute some or all of the steps in the computer program or method involved in the various embodiments of this application, or any combination of the steps mentioned therein.

[0093] To further verify the segmentation accuracy proposed in this invention, further experimental analysis was conducted.

[0094] In traditional methods, the ST-GCN model can only determine the type of motion within a unit of time, and cannot perform fine-grained segmentation of action transition times based on frames. See also Figure 3 This invention achieves segmentation functionality by segmenting the Head and modifying the hyperparameters of the ST-GCN model. See also... Figure 4 To improve the backbone part of the ST-GCN segmentation model, a new Segment GCN is proposed, which allows the model to acquire more spatiotemporal scale information features. The experimental results of the rope skipping motion segmentation and ablation experiment are shown in Table 1.

[0095]

[0096] Table 1

[0097] As shown in Table 1, the rope skipping motion segmentation ablation experiment demonstrates, the Segment GCN motion segmentation model proposed in this invention is preferred, as it can achieve high-precision motion segmentation compared to existing spatiotemporal graph convolutional neural networks.

[0098] It should be noted that although the above embodiments have been described herein, this does not limit the scope of patent protection for this invention. Therefore, any changes and modifications made to the embodiments described herein based on the innovative concept of this invention, or equivalent structural or procedural transformations made using the description and drawings of this invention, directly or indirectly applying the above technical solutions to other related technical fields, are all included within the scope of patent protection for this invention.

Claims

1. A method for constructing a skipping rope recognition network model based on spatiotemporal graph convolution segmentation, characterized in that, The method comprises the following steps: Collecting a video data sample; Performing 3D human key point detection on the video data sample to obtain a sample human key point time sequence S; Frame segmentation and labeling are performed on the sample human key point time sequence S, and the result is recorded as a label L of training data, which comprises: performing curve operation on the human key points in the labeled human key point time sequence to obtain a motion feature curve; finding local maximum and minimum values of the motion feature curve by using a sliding window to obtain a plurality of segmentation intervals of the labeled human key point time sequence, and labeling the plurality of segmentation intervals; the labeling comprises one of a jump state, a falling state, a rope interruption state and other abnormal states; A spatio-temporal graph convolutional neural network is established, which comprises a backbone part and a segmentation Head part, the backbone part is used for spatio-temporal feature extraction, and the segmentation Head part performs frame segmentation on the spatio-temporal feature extraction; The sample human key point time sequence S is used as training data to train the spatio-temporal graph convolutional neural network; a loss function is defined, the parameters of the spatio-temporal graph convolutional neural network are updated through back propagation, and a skipping rope recognition network model is obtained.

2. The method of claim 1, wherein the method is characterized by, The step of collecting the video data sample comprises the following steps: n auxiliary cameras are arranged in front of the jumping rope position c ​ n c The n c Video data samples of the n c auxiliary cameras are obtained through synchronized timestamps.

3. The method of claim 1, wherein the method further comprises: The step of performing frame segmentation and labeling on the sample human key point time sequence S and recording the result as a label L of training data comprises the following steps: A calibration camera is arranged directly in front of the skipping rope movement position; The calibration camera collects calibration video data, and the calibration video data is collected synchronously with the video data sample; 3D human key point detection is performed on the calibration video data to obtain a calibration human key point time sequence; Curve operation is performed on the human key points in the calibration human key point time sequence to obtain a motion feature curve; Local maximum and minimum values of the motion feature curve are found by using a sliding window to obtain a plurality of segmentation intervals of the calibration human key point time sequence, and the plurality of segmentation intervals are labeled; According to the synchronization time stamp and the plurality of segmentation intervals and the labeling of the calibration human key point time sequence, frame segmentation and labeling are performed on the sample human key point time sequence S, and the result is recorded as a label L of training data.

4. The method of claim 1, wherein the method further comprises: The backbone part comprises an STGC module, the STGC module is used for spatio-temporal feature extraction and outputs a feature map; the segmentation Head part comprises a Permute component and an AveragePool2D component, the Permute component performs dimension conversion on the feature map output by the backbone part, and the AveragePool2D component performs two-dimensional average pooling on the output of the Permute component.

5. The method of claim 4, wherein the method further comprises: The backbone part further comprises an Upsample component and a Concat component, the Upsample component performs bilinear interpolation on the output feature map, and up-samples it in the time and space dimensions to obtain a high-resolution feature map; the Concat component stacks the high-resolution feature maps of different scales in the channel dimension to perform multi-scale information fusion. The segmentation Head part further comprises filtering the average-pooled feature map using an NMC method, and taking the class with the highest confidence as the segmentation result output.

6. The method of claim 1, wherein the method is characterized by: The loss function is a segment coupling loss function and a cross-entropy loss function fusion, then where S end represents the real end frame of each sub-action segment, represents the predicted end frame of each sub-action segment, S strat represents the real start frame of each sub-action segment, represents the predicted start frame of each sub-action segment; L i represents the i-th element of the label L of the training data, represents the i-th element of the label predicted by the model.

7. A skipping recognition method based on spatio-temporal graph convolution segmentation, applying the skipping recognition network model generated by the construction method of the skipping recognition network model based on spatio-temporal graph convolution segmentation in any one of claims 1-6. The method comprises the following steps: Obtaining video data; Performing 3D human key point detection on the video data to obtain a human key point time sequence; Inputting the human key point time sequence into a skipping rope recognition network model for frame segmentation and labeling to obtain a skipping rope state; Counting each skipping rope state to obtain the number of times of a jumping state, a falling state, a rope interruption state and other abnormal states.

8. Storage medium, characterized in that: The storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the skipping rope recognition method based on spatio-temporal graph convolution segmentation according to claim 7.

9. An electronic device, characterized by: The computer program is stored on the storage medium and is executed by the processor to realize the steps of the skipping rope recognition method based on spatio-temporal graph convolution segmentation according to claim 7.

Images