Skipping rope automatic counting method and system based on visual and voice collaborative recognition

Abstract

This invention discloses an automatic jump rope counting method and system based on visual and speech collaborative recognition. The method simultaneously acquires video and audio streams from the jumper; extracts the Y-coordinate trajectory data of both ankles from the video stream, and determines the reliability of trajectory points through background interference identification, followed by adaptive filtering to obtain a smooth sequence of ankle movement trajectories; extracts spectral features from the audio stream and identifies key audio events. The identified key audio points are time-aligned with the footwork cycles analyzed from the trajectory sequence, and through multimodal information fusion verification, the trajectory peaks representing valid jumps are accurately identified. Finally, automatic counting is completed based on the number of peaks. This invention integrates visual and speech information, mutually correcting each other, effectively overcoming interference from complex environments, and significantly improving the accuracy and reliability of the counting results.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and sports counting technology, and in particular to an automatic rope skipping counting method and system based on visual and voice collaborative recognition. Background Technology

[0002] Traditional jump rope counting relies mainly on manual labor, which suffers from problems such as high subjectivity, easy fatigue, and difficulty in long-term accurate recording. With the development of computer vision and audio processing technologies, automatic counting methods based on single sensor modes have emerged.

[0003] In existing technologies, video analytics-based counting methods primarily identify jump cycles by detecting changes in human motion trajectories or postures. For example, counting is achieved by extracting trajectory peaks through tracking the vertical displacement of the ankles or head. However, in complex backgrounds, scenes with multiple people or varying lighting conditions, these methods are prone to increased noise in the trajectory data due to target occlusion, tracking loss, or false detection, leading to missed or incorrect counts. Furthermore, relying solely on visual information makes it difficult to distinguish between valid take-offs and landings and invalid movements such as minor everyday swaying.

[0004] On the other hand, audio analysis-based counting methods primarily utilize the impact sound characteristics generated when a jump rope lands for event detection. While these methods can effectively capture the landing moment, they cannot identify the sound source and are easily affected by environmental noise, the sound of others jumping rope, or conversations, leading to false triggers. Furthermore, audio information cannot directly reflect the complete jumping posture, making it difficult to determine whether the movement is correct or whether there are invalid behaviors such as "empty jumps."

[0005] Therefore, there is an urgent need for a new method that can deeply integrate visual and speech information and effectively improve the accuracy and reliability of rope skipping counting in complex environments through collaborative recognition and mutual verification mechanisms. Summary of the Invention

[0006] To address at least one of the aforementioned technical problems, this invention proposes an automatic rope skipping counting method and system based on visual and speech collaborative recognition.

[0007] The first aspect of this invention provides an automatic jump rope counting method based on visual and speech co-recognition, comprising:

[0008] The system acquires a jump rope video stream and a synchronously acquired audio stream. It performs human key point detection on the video stream and extracts the Y-coordinate trajectory data of the two ankles in consecutive frames. At the same time, it performs a short-time Fourier transform on the audio stream to generate a speech spectrum feature vector.

[0009] Background interference is identified in the jump rope video stream. The reliability of the trajectory point at each time point in the Y-coordinate trajectory data is determined based on the background interference identification. The Y-coordinate trajectory data is then subjected to Gaussian low-pass filtering noise processing based on the reliability of the trajectory points to form the Y-trajectory sequence of the jumper's two ankle movements.

[0010] Key audio points are identified based on the speech spectrum feature vector, and the key audio points include the start audio point, the foot landing audio point, and the end audio point.

[0011] The jumper's footwork cycle is identified based on the bi-ankle movement Y-trajectory sequence. The key audio points and the bi-ankle movement cycle are time-aligned to identify the effective jump peaks of the bi-ankle movement Y-trajectory sequence.

[0012] The effective number of jumps for the jumper is calculated based on the effective jump peak.

[0013] In this solution, the process of acquiring the jump rope video stream and simultaneously acquiring the audio stream involves performing human key point detection on the video stream, extracting the Y-coordinate trajectory data of the ankles in consecutive frames, and simultaneously performing a short-time Fourier transform on the audio stream to generate a speech spectrum feature vector. Specifically:

[0014] The video and audio streams of the jump rope jumper are acquired synchronously. Multi-scale feature maps of each frame of the video stream are extracted based on the target detection algorithm. Human boundary features are fused using a path aggregation network to output the human bounding box prediction results. The non-maximum suppression algorithm is used to remove redundant detection boxes to obtain the human detection bounding box of the jump rope jumper in each frame of the image.

[0015] The human body detection bounding boxes in consecutive frames are tracked, the appearance features of the human body detection bounding boxes are extracted, the position state of the human body in the next frame is predicted using Kalman filtering, the predicted position is associated and matched with the detection result of the current frame using the Hungarian algorithm, a unique identifier is assigned to each jumper, and a sequence of human body movement trajectories is constructed.

[0016] A human skeleton key point detection network is constructed, the human detection boundary is input into the human skeleton key point detection network, the human skeleton key points of each frame image in the video stream are identified, and the coordinates of the two-dimensional key points of the human skeleton are output.

[0017] Based on the coordinates of the two-dimensional key points of the human skeleton, the two-dimensional coordinate information of the key points of the two ankles is extracted. Based on the human motion trajectory sequence, the coordinates of the key points of the two ankles in the continuous frames are smoothed to construct the Y-coordinate trajectory data of the two ankles.

[0018] The audio stream is divided into multiple short audio frames. A Hanning window function is applied to each audio frame for windowing. A fast Fourier transform is performed on each windowed audio frame to convert the time-domain audio signal into a frequency-domain signal. The amplitude spectrum of each frequency-domain signal is then calculated.

[0019] Based on the amplitude spectrum, Mel frequency cepstral coefficients of each audio frame are extracted to form a Mel spectrum feature vector. The Mel spectrum feature vectors are arranged in chronological order to construct a speech spectrum feature matrix.

[0020] The speech spectrum feature matrix is ​​subjected to first-order and second-order time-difference calculations to obtain dynamic spectrum features. The Mel spectrum features are then concatenated with the dynamic spectrum features to generate a speech spectrum feature vector.

[0021] In this solution, the process of identifying background interference in the jump rope video stream, determining the reliability of trajectory points at each time point in the Y-coordinate trajectory data based on the identified background interference, and then applying Gaussian low-pass filtering noise processing to the Y-coordinate trajectory data based on the reliability of the trajectory points to construct the jumper's bi-ankle movement Y-trajectory sequence, specifically involves:

[0022] The union region of the human detection bounding boxes in each frame of the jump rope video stream is obtained as the background analysis region. The average area of ​​the human bounding boxes and the area of ​​the union region in the video stream are calculated. The stability coefficient of the background region change is determined based on the ratio of the average area to the area of ​​the union region.

[0023] For each video frame within the background analysis area, the dense optical flow method is used to calculate the motion vectors of all pixels between the current frame and the previous frame. The amplitude of all motion vectors is counted, and the mean amplitude and variance of the motion vectors within the background analysis area are calculated. The background dynamic interference intensity of each video frame within the background analysis area is determined based on the mean amplitude and variance of the amplitude.

[0024] The reliability of trajectory points at each time point in the bi-ankle Y-coordinate trajectory data is evaluated based on the background region change stability coefficient and the background dynamic interference intensity.

[0025] The initial confidence values ​​of trajectory points in a continuous frame sequence are smoothed based on the Hidden Markov Model. The confidence value of the trajectory point at each time point is used as the observation state of the Hidden Markov Model. The confidence sequence of trajectory points is obtained by decoding with the Viterbi algorithm.

[0026] Background noise filtering is performed on the Y-coordinate trajectory data based on Gaussian low-pass filtering. The filtering intensity of each trajectory point in the Y-coordinate trajectory data is determined according to the trajectory point confidence sequence. The filtering parameters of the Gaussian low-pass filter are determined according to the filtering intensity. The filtering parameters include the filter kernel size and standard deviation.

[0027] The background noise of the Y-coordinate trajectory data is filtered according to the filtering parameters to obtain the Y-trajectory sequence of the two ankle movements.

[0028] In this solution, the identification of key audio points based on the speech spectrum feature vector, wherein the key audio points include the start audio point, the foot landing audio point, and the end audio point, specifically:

[0029] Acquire historical jump rope audio data, extract speech domain feature vectors from the historical jump rope audio data, obtain the collection time period of key audio points in the historical jump rope audio data, and extract historical speech domain feature vectors of key audio points based on the collection time period.

[0030] The historical audio domain feature vector of each key audio point is labeled with the name of the key audio point to form a training dataset.

[0031] A key audio point recognition model is constructed based on a deep neural network. The model includes an input layer, a hidden layer, and an output layer. The key audio point recognition model is trained using the training dataset. The network weight parameters are optimized by minimizing the cross-entropy loss function through the backpropagation algorithm.

[0032] The real-time acquired speech spectrum feature vector is input into the trained key audio point recognition model to obtain the probability distribution of key audio points in each audio frame;

[0033] Based on the probability distribution, key audio points in the speech spectrum feature vector are identified, and key audio point type data is output.

[0034] In this solution, the step of identifying the jumper's footwork cycle based on the Y-trajectory sequence of the double ankle movement, and performing time alignment between the key audio points and the footwork cycle to identify the effective jump peaks of the Y-trajectory sequence of the double ankle movement, specifically involves:

[0035] Based on the sliding time window, local maxima and local minima are detected in the Y-trajectory sequence of the bi-ankle movement. Adjacent maxima and minima are combined to form candidate peaks and candidate troughs. The amplitude difference between consecutive candidate peaks and candidate troughs is calculated as the jump amplitude, and the time interval between consecutive candidate peaks is calculated as the jump period.

[0036] Based on the jump amplitude and jump cycle, a preliminary dual-gait gait cycle sequence is constructed. Based on the identified start and end audio points, a gait cycle segment located after the start audio point and before the end audio point is extracted from the dual-gait gait cycle sequence as the cycle segment to be analyzed.

[0037] The audio point of each foot landing in the period segment to be analyzed is matched with the candidate trough that is closest in time. If the match is successful, the candidate trough is marked as the confirmed landing point. The Y trajectory subsequence within the preset time window before and after the confirmed landing point is extracted. The waveform features of the Y trajectory subsequence are calculated. The waveform features include the steepness of the peak, the flatness of the trough, and the smoothness of the trajectory between the peak and the trough.

[0038] Based on the waveform features corresponding to all confirmed landing points, a clustering algorithm is used to generate waveform feature templates for normal jumps, which include take-off and periodic cyclic jumps.

[0039] The similarity of the Y trajectory subsequences corresponding to all candidate peaks within the period segment to be analyzed with the waveform feature template of normal jumping is calculated. If the similarity is lower than a preset threshold, the Y trajectory subsequences corresponding to the candidate peaks are determined to be interference sequences caused by rope skipping errors and are removed.

[0040] Candidate peaks with similarity higher than a preset threshold are retained and output as valid skip peak sequences.

[0041] In this scheme, the calculation of the effective jump count based on the effective jump peak is specifically as follows:

[0042] Extract the timestamp information corresponding to each valid jump peak, sort the valid jump peaks by time according to the timestamp information, and construct a time series of valid jump peaks.

[0043] Initialize the effective jump counter and set the jump continuity determination time window. Calculate the time interval between adjacent effective jump peaks in the effective jump peak time sequence and determine whether the time interval between adjacent effective jump peaks is less than the jump continuity determination time window.

[0044] When the time interval between adjacent valid jump peaks is less than the jump continuity determination time window, the adjacent valid jump peaks are determined to belong to the same continuous jump sequence, and the adjacent valid jump peaks are merged into the same continuous jump sequence.

[0045] When the time interval between adjacent valid jump peaks is not less than the jump continuity determination time window, the current continuous jump sequence is determined to have ended, and a new continuous jump sequence is recorded.

[0046] Calculate the number of valid jump peaks in each consecutive jump sequence, and sum the number of valid jump peaks in all valid jump sequences to obtain the total number of valid jumps for the jumper.

[0047] A second aspect of the present invention also provides an automatic rope skipping counting system based on visual and speech co-recognition. The system includes a memory and a processor. The memory includes a method program for automatic rope skipping counting based on visual and speech co-recognition. When the processor executes the automatic rope skipping counting method program based on visual and speech co-recognition, it performs the following steps:

[0048] The system acquires a jump rope video stream and a synchronously acquired audio stream. It performs human key point detection on the video stream and extracts the Y-coordinate trajectory data of the two ankles in consecutive frames. At the same time, it performs a short-time Fourier transform on the audio stream to generate a speech spectrum feature vector.

[0049] Background interference is identified in the jump rope video stream. The reliability of the trajectory point at each time point in the Y-coordinate trajectory data is determined based on the background interference identification. The Y-coordinate trajectory data is then subjected to Gaussian low-pass filtering noise processing based on the reliability of the trajectory points to form the Y-trajectory sequence of the jumper's two ankle movements.

[0050] Key audio points are identified based on the speech spectrum feature vector, and the key audio points include the start audio point, the foot landing audio point, and the end audio point.

[0051] The jumper's footwork cycle is identified based on the bi-ankle movement Y-trajectory sequence. The key audio points and the bi-ankle movement cycle are time-aligned to identify the effective jump peaks of the bi-ankle movement Y-trajectory sequence.

[0052] The effective number of jumps for the jumper is calculated based on the effective jump peak.

[0053] This invention discloses an automatic jump rope counting method and system based on visual and speech collaborative recognition. The method simultaneously acquires video and audio streams from the jumper; extracts the Y-coordinate trajectory data of both ankles from the video stream, and determines the reliability of trajectory points through background interference identification, followed by adaptive filtering to obtain a smooth sequence of ankle movement trajectories; extracts spectral features from the audio stream and identifies key audio events. The identified key audio points are time-aligned with the footwork cycles analyzed from the trajectory sequence, and through multimodal information fusion verification, the trajectory peaks representing valid jumps are accurately identified. Finally, automatic counting is completed based on the number of peaks. This invention integrates visual and speech information, mutually correcting each other, effectively overcoming interference from complex environments, and significantly improving the accuracy and reliability of the counting results. Attached Figure Description

[0054] Figure 1 A flowchart of an automatic rope skipping counting method based on visual and speech collaborative recognition according to the present invention is shown;

[0055] Figure 2The flowchart illustrating the present invention for identifying key audio points based on speech spectral feature vectors is shown.

[0056] Figure 3 A flowchart illustrating the present invention for identifying effective jump peaks in a sequence of Y-trajectories of bi-ankle movements is shown.

[0057] Figure 4 A block diagram of an automatic jump rope counting system based on visual and speech collaborative recognition according to the present invention is shown.

[0058] Figure 5 The diagram shows the key points of the human skeleton according to the present invention. Detailed Implementation

[0059] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0060] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0061] Figure 1 The flowchart of an automatic rope skipping counting method based on visual and speech co-recognition according to the present invention is shown.

[0062] like Figure 1 As shown, the first aspect of the present invention provides an automatic rope skipping counting method based on visual and speech collaborative recognition, comprising:

[0063] S102, acquire the rope skipping video stream and synchronously acquired audio stream of the rope skipper, perform human key point detection on the video stream, extract the Y coordinate trajectory data of the two ankles in continuous frames, and at the same time perform short-time Fourier transform on the audio stream to generate speech spectrum feature vector.

[0064] S104, perform background interference identification on the jump rope video stream, determine the reliability of the trajectory point at each time point in the Y coordinate trajectory data based on the background interference identification, and perform Gaussian low-pass filtering noise processing on the Y coordinate trajectory data based on the reliability of the trajectory point to form the Y trajectory sequence of the jump rope user's two ankle movements.

[0065] S106, Identify key audio points based on the speech spectrum feature vector, the key audio points including the start audio point, the foot landing audio point, and the end audio point;

[0066] S108, Identify the jumper's footwork cycle based on the bi-ankle movement Y-trajectory sequence, perform time alignment operation on the key audio points and the bi-ankle movement cycle, and identify the effective jump peak of the bi-ankle movement Y-trajectory sequence;

[0067] S110, calculate the number of effective jumps by the jumper based on the effective jump peak.

[0068] It should be noted that the process involves acquiring synchronized video and audio streams, detecting human key points in the video stream to extract the Y-coordinate trajectory of both ankles, and simultaneously generating a speech spectrum feature vector. Background interference is identified in the video stream, and a confidence level is assigned to each time point in the trajectory based on this. Then, the Gaussian low-pass filter parameters are adaptively adjusted based on this confidence level to selectively suppress noise components while preserving high-frequency transients of the real motion. This prevents attenuation or displacement of jump peaks caused by uniform smoothing, improving the temporal stability and integrity of the trajectory in complex scenarios. Key audio points such as start, foot landing, and end are identified using the speech spectrum feature vector. The Y-trajectory sequence of both ankles is used for gait cycle recognition and finely time-aligned with the speech key points. This not only eliminates misjudgments caused by phase drift or tracking delay but also filters out spurious peaks caused by non-jumping movements through acoustic-visual consistency, accurately locating the true take-off and landing moments. Based on the identification of valid jump peaks, waveform feature templates are constructed, and candidate peaks are judged for similarity. This distinguishes normal periodic jumps from abnormal sequences caused by errors or environmental interference at the feature level, effectively reducing miscounting and undercounting.

[0069] According to an embodiment of the present invention, the process of acquiring the rope-jumping video stream and the synchronously acquired audio stream, performing human key point detection on the video stream, extracting the Y-coordinate trajectory data of the two ankles in consecutive frames, and simultaneously performing a short-time Fourier transform on the audio stream to generate a speech spectrum feature vector, specifically includes:

[0070] The video and audio streams of the jump rope jumper are acquired synchronously. Multi-scale feature maps of each frame of the video stream are extracted based on the target detection algorithm. Human boundary features are fused using a path aggregation network to output the human bounding box prediction results. The non-maximum suppression algorithm is used to remove redundant detection boxes to obtain the human detection bounding box of the jump rope jumper in each frame of the image.

[0071] The human body detection bounding boxes in consecutive frames are tracked, the appearance features of the human body detection bounding boxes are extracted, the position state of the human body in the next frame is predicted using Kalman filtering, the predicted position is associated and matched with the detection result of the current frame using the Hungarian algorithm, a unique identifier is assigned to each jumper, and a sequence of human body movement trajectories is constructed.

[0072] A human skeleton key point detection network is constructed, the human detection boundary is input into the human skeleton key point detection network, the human skeleton key points of each frame image in the video stream are identified, and the coordinates of the two-dimensional key points of the human skeleton are output.

[0073] Based on the coordinates of the two-dimensional key points of the human skeleton, the two-dimensional coordinate information of the key points of the two ankles is extracted. Based on the human motion trajectory sequence, the coordinates of the key points of the two ankles in the continuous frames are smoothed to construct the Y-coordinate trajectory data of the two ankles.

[0074] The audio stream is divided into multiple short audio frames. A Hanning window function is applied to each audio frame for windowing. A fast Fourier transform is performed on each windowed audio frame to convert the time-domain audio signal into a frequency-domain signal. The amplitude spectrum of each frequency-domain signal is then calculated.

[0075] Based on the amplitude spectrum, Mel frequency cepstral coefficients of each audio frame are extracted to form a Mel spectrum feature vector. The Mel spectrum feature vectors are arranged in chronological order to construct a speech spectrum feature matrix.

[0076] The speech spectrum feature matrix is ​​subjected to first-order and second-order time-difference calculations to obtain dynamic spectrum features. The Mel spectrum features are then concatenated with the dynamic spectrum features to generate a speech spectrum feature vector.

[0077] It should be noted that the target detection algorithm includes, but is not limited to, YOLOv7. Utilizing YOLOv7's multi-scale feature extraction and non-maximum suppression mechanism, it can stably identify the human body boundary of a jumper in complex backgrounds and under different lighting conditions, thus ensuring the spatial accuracy and target integrity of human detection. Combined with DeepSORT for target tracking, appearance feature matching, and Kalman filter prediction, it achieves continuous tracking of human identity across frames, effectively preventing detection interruptions caused by occlusion or high-speed movement of the jumper. A human skeleton keypoint detection network is constructed using the HRNet high-resolution pose estimation model to extract key points of the human skeleton, further obtaining the precise two-dimensional coordinates of both ankles. Combined with the motion trajectory sequence, smoothing processing is performed so that the Y-coordinate trajectory of both ankles can realistically reflect the periodic changes in the foot's vertical movement, exhibiting strong temporal continuity and anti-shake performance. In terms of speech processing, a short-time Fourier transform combined with Hanning window weighting is used to effectively reduce edge effects and preserve the time-frequency characteristics of the rope landing sound. By extracting the Mel frequency cepstral coefficients and their first and second order differences, a high-dimensional speech spectrum feature vector that can describe both energy distribution and change trends is obtained, thereby achieving a fine-grained characterization of the acoustic events of rope jumping. Figure 5 This invention illustrates a diagram of key points on the human skeleton, including:

[0078] ;

[0079] According to an embodiment of the present invention, the step of identifying background interference in the jump rope video stream, determining the reliability of trajectory points at each time point in the Y-coordinate trajectory data based on the identified background interference, and performing Gaussian low-pass filtering noise processing on the Y-coordinate trajectory data based on the reliability of the trajectory points to construct a Y-trajectory sequence of the jumper's two ankle movements, specifically includes:

[0080] The union region of the human detection bounding boxes in each frame of the jump rope video stream is obtained as the background analysis region. The average area of ​​the human bounding boxes and the area of ​​the union region in the video stream are calculated. The stability coefficient of the background region change is determined based on the ratio of the average area to the area of ​​the union region.

[0081] For each video frame within the background analysis area, the dense optical flow method is used to calculate the motion vectors of all pixels between the current frame and the previous frame. The amplitude of all motion vectors is counted, and the mean amplitude and variance of the motion vectors within the background analysis area are calculated. The background dynamic interference intensity of each video frame within the background analysis area is determined based on the mean amplitude and variance of the amplitude.

[0082] The reliability of trajectory points at each time point in the bi-ankle Y-coordinate trajectory data is evaluated based on the background region change stability coefficient and the background dynamic interference intensity.

[0083] The initial confidence values ​​of trajectory points in a continuous frame sequence are smoothed based on the Hidden Markov Model. The confidence value of the trajectory point at each time point is used as the observation state of the Hidden Markov Model. The confidence sequence of trajectory points is obtained by decoding with the Viterbi algorithm.

[0084] Background noise filtering is performed on the Y-coordinate trajectory data based on Gaussian low-pass filtering. The filtering intensity of each trajectory point in the Y-coordinate trajectory data is determined according to the trajectory point confidence sequence. The filtering parameters of the Gaussian low-pass filter are determined according to the filtering intensity. The filtering parameters include the filter kernel size and standard deviation.

[0085] The background noise of the Y-coordinate trajectory data is filtered according to the filtering parameters to obtain the Y-trajectory sequence of the two ankle movements.

[0086] It should be noted that during rope skipping motion recognition, video signals are often affected by complex backgrounds, lighting changes, motion blur, and interference from other moving targets. This can lead to jitter, drift, or false detections in the detection results of human key points, resulting in a large amount of noise caused by non-human motion mixed into the Y-coordinate trajectory signal of both ankles. Consequently, this affects the stability of jump cycle recognition and the accuracy of jump count statistics. Therefore, by analyzing the area change of the union region of the human bounding box to obtain the background stability coefficient, and combining it with the mean and variance of the motion vector amplitude calculated by dense optical flow, the dynamic interference intensity of the video background can be quantitatively described, thereby identifying potential non-human motion factors in the background. Subsequently, the reliability of the trajectory points is temporally smoothed based on a Hidden Markov Model, making the trajectory reliability distribution more consistent with the changes in the real motion state and avoiding the destruction of the overall trajectory quality by short-term false detections. On this basis, a reliability-driven Gaussian low-pass filter is used to achieve adaptive adjustment of the filter parameters, so that more motion details are retained in the high reliability section, while the smoothing suppression is strengthened in the low reliability section. This effectively eliminates the pseudo-fluctuations caused by background interference while ensuring the integrity of the jump signal.

[0087] Figure 2 The flowchart of the present invention for identifying key audio points based on speech spectrum feature vectors is shown.

[0088] According to an embodiment of the present invention, the step of identifying key audio points based on the speech spectrum feature vector, wherein the key audio points include a start audio point, a foot landing audio point, and an end audio point, specifically:

[0089] Acquire historical jump rope audio data, extract speech domain feature vectors from the historical jump rope audio data, obtain the collection time period of key audio points in the historical jump rope audio data, and extract historical speech domain feature vectors of key audio points based on the collection time period.

[0090] The historical audio domain feature vector of each key audio point is labeled with the name of the key audio point to form a training dataset.

[0091] A key audio point recognition model is constructed based on a deep neural network. The model includes an input layer, a hidden layer, and an output layer. The key audio point recognition model is trained using the training dataset. The network weight parameters are optimized by minimizing the cross-entropy loss function through the backpropagation algorithm.

[0092] The real-time acquired speech spectrum feature vector is input into the trained key audio point recognition model to obtain the probability distribution of key audio points in each audio frame;

[0093] Based on the probability distribution, key audio points in the speech spectrum feature vector are identified, and key audio point type data is output.

[0094] It should be noted that in the automatic counting process of rope skipping, relying solely on visual signals is often affected by factors such as changes in lighting, occlusion, and camera angle deviations, leading to delays or misjudgments in action recognition. Each jump and landing in rope skipping is accompanied by significant sound characteristics. Therefore, identifying key audio points can provide independent and timely acoustic evidence for visual recognition, enabling cross-modal action event calibration and compensation. By extracting and identifying the start audio point, foot landing audio point, and end audio point, the start and end boundaries of the rope skipping action in the time dimension can be accurately characterized, eliminating phase drift problems in visual detection and maintaining continuous monitoring of jump events even when visual features are missing. This claim employs a key audio point recognition model based on a deep neural network. Utilizing its nonlinear mapping and pattern abstraction capabilities in a high-dimensional feature space, it can automatically learn subtle differences between different audio events from the speech spectrum in complex noisy environments, significantly improving the ability to distinguish footstep impact sounds from environmental noise. The model, optimized through backpropagation training, can quickly output the probability distribution of key audio points in the real-time audio stream, achieving accurate identification of the temporal nodes of the rope skipping action. The number of nodes in the input layer is consistent with the dimension of the speech spectrum feature vector. Each hidden layer consists of a fully connected layer and a ReLU activation function. The output layer uses the Softmax activation function and outputs the classification probabilities of three key audio points.

[0095] Figure 3 A flowchart illustrating the present invention for identifying effective jump peaks in a sequence of Y-trajectories of bi-ankle movements is shown.

[0096] According to an embodiment of the present invention, the step of identifying the jumper's footwork cycle based on the Y-track sequence of the double ankle movement, performing time alignment between the key audio points and the footwork cycle, and identifying the effective jump peaks of the Y-track sequence of the double ankle movement specifically involves:

[0097] Based on the sliding time window, local maxima and local minima are detected in the Y-trajectory sequence of the bi-ankle movement. Adjacent maxima and minima are combined to form candidate peaks and candidate troughs. The amplitude difference between consecutive candidate peaks and candidate troughs is calculated as the jump amplitude, and the time interval between consecutive candidate peaks is calculated as the jump period.

[0098] Based on the jump amplitude and jump cycle, a preliminary dual-gait gait cycle sequence is constructed. Based on the identified start and end audio points, a gait cycle segment located after the start audio point and before the end audio point is extracted from the dual-gait gait cycle sequence as the cycle segment to be analyzed.

[0099] The audio point of each foot landing in the period segment to be analyzed is matched with the candidate trough that is closest in time. If the match is successful, the candidate trough is marked as the confirmed landing point. The Y trajectory subsequence within the preset time window before and after the confirmed landing point is extracted. The waveform features of the Y trajectory subsequence are calculated. The waveform features include the steepness of the peak, the flatness of the trough, and the smoothness of the trajectory between the peak and the trough.

[0100] Based on the waveform features corresponding to all confirmed landing points, a clustering algorithm is used to generate waveform feature templates for normal jumps, which include take-off and periodic cyclic jumps.

[0101] The similarity of the Y trajectory subsequences corresponding to all candidate peaks within the period segment to be analyzed with the waveform feature template of normal jumping is calculated. If the similarity is lower than a preset threshold, the Y trajectory subsequences corresponding to the candidate peaks are determined to be interference sequences caused by rope skipping errors and are removed.

[0102] Candidate peaks with similarity higher than a preset threshold are retained and output as valid skip peak sequences.

[0103] It should be noted that the trajectory of a jumper's feet includes not only valid peak signals from actual jumps but also spurious peak signals caused by jump errors. Directly using all trajectory peaks for counting would lead to an inflated jump count or distorted rhythm analysis. Therefore, it is essential to screen and identify candidate peaks in the Y-coordinate trajectory of the ankles, extracting the valid jump peak sequence that truly reflects the complete take-off-landing cycle. By detecting local extrema within a sliding time window to construct a gait cycle and combining this with key audio points obtained from speech recognition for time alignment, synchronous verification of visual motion signals and acoustic events is achieved, effectively eliminating detection delay and phase drift issues. Subsequently, by extracting and confirming the trajectory waveform features near the landing point and using a clustering algorithm to establish a normal jump template, stable jumps and abnormal interference signals can be distinguished in the feature space. A template similarity discrimination mechanism is used to screen candidate peaks, ensuring that the output valid jump peak sequence retains only trajectory patterns highly consistent with the periodic jumping behavior of the human body. Because the jump rope speed is too fast during the counting process, the jumper will still maintain the same jumping cycle state as before the end when they hear the end audio point. Therefore, this invention does not need to study the jumping waveform when the feet land at the end of the jump rope.

[0104] According to an embodiment of the present invention, the step of calculating the effective number of jumps by the jumper based on the effective jump peak specifically involves:

[0105] Extract the timestamp information corresponding to each valid jump peak, sort the valid jump peaks by time according to the timestamp information, and construct a time series of valid jump peaks.

[0106] Initialize the effective jump counter and set the jump continuity determination time window. Calculate the time interval between adjacent effective jump peaks in the effective jump peak time sequence and determine whether the time interval between adjacent effective jump peaks is less than the jump continuity determination time window.

[0107] When the time interval between adjacent valid jump peaks is less than the jump continuity determination time window, the adjacent valid jump peaks are determined to belong to the same continuous jump sequence, and the adjacent valid jump peaks are merged into the same continuous jump sequence.

[0108] When the time interval between adjacent valid jump peaks is not less than the jump continuity determination time window, the current continuous jump sequence is determined to have ended, and a new continuous jump sequence is recorded.

[0109] Calculate the number of valid jump peaks in each consecutive jump sequence, and sum the number of valid jump peaks in all valid jump sequences to obtain the total number of valid jumps for the jumper.

[0110] It should be noted that by extracting the timestamp of each valid peak and sorting them by time, a jump continuity determination time window is introduced, enabling the system to distinguish between continuous jumps and interrupted jumps. This effectively identifies pauses, errors, or rhythm changes during movement, preventing miscounts caused by occasional jitters or non-jumping actions. By dynamically grouping the time intervals of adjacent peaks, the system can adapt to different jump rope speeds and rhythms, maintaining the counting algorithm's adaptability to individual differences. Furthermore, by accumulating the peaks within a continuous jump sequence and globally integrating the results from different sequences, an accurate total number of valid jumps can be obtained.

[0111] According to an embodiment of the present invention, it further includes:

[0112] A continuity analysis is performed on the Y-trajectory sequence of bi-ankle movement. By calculating the difference in Y coordinates between adjacent frames and setting a continuity threshold, the broken intervals in the trajectory sequence are identified. The broken intervals include missing or invalid Y coordinate data in multiple consecutive frames.

[0113] Extract the speech spectrum feature vector sub-segment corresponding to the time period of the break interval from the audio stream, and denote it as the audio segment of the break interval;

[0114] Within the audio time period corresponding to the complete trajectory segment before and after the fracture interval, multiple speech spectrum feature vector reference segments with the same length as the audio segment of the fracture interval are slidably extracted, and the feature similarity between each reference segment and the audio segment of the fracture interval is calculated. The top K reference segments with the highest similarity are selected as the best matching audio segments.

[0115] Obtain the video frame corresponding to the time period of each best-matched audio segment, and extract the Y-coordinate data of the two ankles of the video frame to form a set of candidate compensation trajectory segments;

[0116] Based on the dynamic time warping algorithm, the morphological matching degree between each candidate compensation trajectory segment and the complete trajectory before and after the break interval is calculated, and the candidate compensation trajectory segment with the highest morphological matching degree is selected as the optimal compensation template.

[0117] Based on the duration of the fracture interval, the optimal compensation template is nonlinearly scaled on the time axis so that its length is completely consistent with the fracture interval. The scaled compensation trajectory segment is then seamlessly inserted into the fracture interval of the original Y trajectory sequence to generate a visually complete bi-ankle movement Y trajectory sequence.

[0118] According to an embodiment of the present invention, selecting the top K reference segments with the highest similarity as the best matching audio segments specifically involves:

[0119] An audio feature matching network based on deep metric learning is used to calculate the feature similarity between the reference segment and the audio segment of the fractured area. This network is based on a pre-trained audio encoder and maps the input audio segment into an embedding vector in a high-dimensional feature space.

[0120] In the high-dimensional feature space, the cosine distance between the embedding vector of the audio segment in the broken region and the embedding vector of each reference segment is calculated, and the K reference segments with the shortest distance are determined as the best matching audio segments.

[0121] The audio feature matching network is trained using a jump rope audio dataset containing positive and negative sample pairs. Positive sample pairs are audio segments from different cycles within the same jump period, while negative sample pairs are audio segments from different jump types or different people. The network parameters are optimized using a triplet loss function, so that audio segments with the same jump pattern cluster in the embedding space, while audio segments with different patterns are far apart from each other.

[0122] After obtaining the best matching audio segment, the Y-coordinate trajectory data of its corresponding time point is extracted. Based on the amplitude and trend of the complete trajectory before and after the break interval, the extracted Y-coordinate trajectory data is normalized and corrected for the trend, so that the candidate compensation trajectory segment can be smoothly connected with the trajectory at both ends of the break interval in terms of shape.

[0123] The corrected candidate compensation trajectory segments are added to the candidate compensation trajectory segment set for subsequent morphological matching degree calculation and optimal compensation template selection.

[0124] It should be noted that when the jumper is briefly obscured or rotates rapidly, causing keypoint detection failure, data breaks occur in the Y-track sequence of the ankle movements. These breaks directly lead to the loss of jump peak information, resulting in count omissions. By utilizing the strong correlation between audio signals (mainly the regular foot landing sound and rope shaking sound) and visual motion trajectories during rope jumping, audio modalities are used as reliable "clues" to guide and reconstruct the broken visual trajectory data. Specifically, through deep metric learning technology, the historical time period most similar to the audio pattern of the broken period is accurately located in the audio feature embedding space. Then, the normal visual trajectory of the corresponding time period is used as a high-quality template. Through dynamic time warping and adaptive scaling, the physical meaning of the broken trajectory is made realistic and the morphological matching degree is high, achieving compensation. This significantly improves the counting integrity of the system in complex scenarios such as partial occlusion, minimizing the missed counts caused by trajectory breaks. Secondly, through audio similarity-based search, the compensated trajectory segments maintain consistency with the user's current jumping style and rhythm, avoiding kinematic distortion that may be caused by simple interpolation and ensuring the accuracy of the counting logic.

[0125] Figure 4 A block diagram of an automatic jump rope counting system based on visual and voice collaborative recognition according to the present invention is shown.

[0126] A second aspect of the present invention also provides an automatic rope skipping counting system based on visual and speech co-recognition. The system includes a memory 401, a processor 402, and a communication interface 403. The memory includes a method program for automatic rope skipping counting based on visual and speech co-recognition. The communication interface is used for data connection and communication between the memory and the processor. When the automatic rope skipping counting method program based on visual and speech co-recognition is executed by the processor, the following steps are implemented:

[0127] The system acquires a jump rope video stream and a synchronously acquired audio stream. It performs human key point detection on the video stream and extracts the Y-coordinate trajectory data of the two ankles in consecutive frames. At the same time, it performs a short-time Fourier transform on the audio stream to generate a speech spectrum feature vector.

[0128] Background interference is identified in the jump rope video stream. The reliability of the trajectory point at each time point in the Y-coordinate trajectory data is determined based on the background interference identification. The Y-coordinate trajectory data is then subjected to Gaussian low-pass filtering noise processing based on the reliability of the trajectory points to form the Y-trajectory sequence of the jumper's two ankle movements.

[0129] Key audio points are identified based on the speech spectrum feature vector, and the key audio points include the start audio point, the foot landing audio point, and the end audio point.

[0130] The jumper's footwork cycle is identified based on the bi-ankle movement Y-trajectory sequence. The key audio points and the bi-ankle movement cycle are time-aligned to identify the effective jump peaks of the bi-ankle movement Y-trajectory sequence.

[0131] The effective number of jumps for the jumper is calculated based on the effective jump peak.

[0132] This invention discloses an automatic jump rope counting method and system based on visual and speech collaborative recognition. The method simultaneously acquires video and audio streams from the jumper; extracts the Y-coordinate trajectory data of both ankles from the video stream, and determines the reliability of trajectory points through background interference identification, followed by adaptive filtering to obtain a smooth sequence of ankle movement trajectories; extracts spectral features from the audio stream and identifies key audio events. The identified key audio points are time-aligned with the footwork cycles analyzed from the trajectory sequence, and through multimodal information fusion verification, the trajectory peaks representing valid jumps are accurately identified. Finally, automatic counting is completed based on the number of peaks. This invention integrates visual and speech information, mutually correcting each other, effectively overcoming interference from complex environments, and significantly improving the accuracy and reliability of the counting results.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0134] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0136] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0138] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A skipping rope automatic counting method based on visual and voice collaborative recognition, characterized in that, The method comprises the following steps: obtaining a skipping video stream of a skipper and a synchronously collected audio stream, performing human key point detection on the video stream, extracting Y coordinate track data of double ankle in continuous frames, simultaneously performing short-time Fourier transform on the audio stream to generate a speech spectrum feature vector; performing background interference recognition on the skipping video stream, determining the track point credibility of each time point in the Y coordinate track data according to the background interference recognition, performing Gaussian low-pass filtering noise processing on the Y coordinate track data according to the track point credibility, and constituting a double ankle movement Y track sequence of the skipper; recognizing key audio points according to the speech spectrum feature vector, wherein the key audio points include a starting audio point, a foot landing audio point and an ending audio point; recognizing a double foot gait cycle of the skipper according to the double ankle movement Y track sequence, performing time alignment operation on the key audio points and the double foot gait cycle, and recognizing an effective jumping wave peak of the double ankle movement Y track sequence, specifically: detecting local maximum values and local minimum values of the double ankle movement Y track sequence based on a sliding time window, combining adjacent maximum values and minimum values to form a candidate wave peak and a candidate wave valley, calculating the amplitude difference between continuous candidate wave peaks and candidate wave valleys as a jumping amplitude, and calculating the time interval between continuous candidate wave peaks as a jumping period; constructing a preliminary double foot gait cycle sequence according to the jumping amplitude and the jumping period, and cutting a gait cycle segment located after the starting audio point and before the ending audio point from the double foot gait cycle sequence as an analysis cycle segment according to the recognized starting audio point and ending audio point; matching each foot landing audio point in the analysis cycle segment with the most adjacent candidate wave valley in time, marking the candidate wave valley as a confirmed landing point if the matching is successful, extracting a Y track sub-sequence in a preset time window before and after the confirmed landing point as a center, and calculating a waveform feature of the Y track sub-sequence, wherein the waveform feature includes a wave peak steepness, a wave valley flatness and a wave peak and wave valley track smoothness; generating a waveform feature template of normal jumping based on the waveform features corresponding to all confirmed landing points, wherein the normal jumping includes jumping, periodic cyclic jumping; performing similarity calculation on the Y track sub-sequence corresponding to each candidate wave peak in the analysis cycle segment and the waveform feature template of normal jumping, and determining that the Y track sub-sequence corresponding to the candidate wave peak is an interference sequence generated by skipping failure and eliminating the interference sequence if the similarity is lower than a preset threshold; retaining the candidate wave peak with a similarity higher than the preset threshold and outputting the candidate wave peak as an effective jumping wave peak sequence; calculating an effective jumping number of the skipper according to the effective jumping wave peak.

2. The method of claim 1, wherein the method is a method of automatically counting a number of jumps of a user using a rope skipping device based on a visual and voice cooperative recognition. The method comprises the following steps: The rope jumper's rope video stream and audio stream data are synchronously acquired, multi-scale feature maps of each frame of image in the video stream are extracted based on a target detection algorithm, human body boundary feature fusion is performed by using a path aggregation network, a human body boundary box prediction result is output, a non-maximum suppression algorithm is used to remove redundant detection boxes, and a human body detection boundary box of each frame of image of the rope jumper is obtained; The human body detection boundary box in the continuous frames is tracked, appearance features of the human body detection boundary box are extracted, a Kalman filter is used to predict the position state of the human body in the next frame, a Hungarian algorithm is used to associate and match the predicted position with the detection result of the current frame, a unique identifier is assigned to each rope jumper, and a human body motion trajectory sequence is constructed; A human body skeleton key point detection network is constructed, the human body detection boundary is input into the human body skeleton key point detection network, human body skeleton key points of each frame of image in the video stream are identified, and human body skeleton two-dimensional key point coordinates are output; Two-dimensional coordinate information of double-ankle key points is extracted according to the human body skeleton two-dimensional key point coordinates, and the double-ankle key point coordinates in the continuous frames are subjected to trajectory smoothing processing based on the human body motion trajectory sequence, and double-ankle Y-coordinate trajectory data are constructed; The audio stream is divided into a plurality of short-period audio frames, a Hanning window function is applied to each audio frame for windowing operation, each windowed audio frame is subjected to fast Fourier transform, a time-domain audio signal is converted into a frequency-domain signal, and the amplitude spectrum of each frequency-domain signal is calculated; Mel-frequency cepstrum coefficients of each audio frame are extracted based on the amplitude spectrum, a mel-frequency spectrum feature vector is constituted, the mel-frequency spectrum feature vector is arranged in time sequence, a speech spectrum feature matrix is constructed, and first-order difference and second-order difference calculations are performed on the speech spectrum feature matrix in the time dimension to obtain dynamic spectrum features. The mel-frequency spectrum features and the dynamic spectrum features are spliced to generate a speech spectrum feature vector. 3.The method of claim 1, wherein, The rope video stream is subjected to background interference recognition, the trajectory point credibility of each time point in the Y-coordinate trajectory data is determined according to the background interference recognition, the Y-coordinate trajectory data is subjected to Gaussian low-pass filter noise processing according to the trajectory point credibility, and a double-ankle motion Y-trajectory sequence of the rope jumper is constituted, specifically as follows: A union region of each human body detection boundary box in the rope video stream is acquired as a background analysis region, the average area of the human body boundary box in the video stream and the area of the union region are calculated, and a background region change stability coefficient is determined according to the ratio of the average area to the area of the union region; A dense optical flow method is used to calculate the motion vectors of all pixel points between the current frame and the previous frame for each frame of video in the background analysis region, the amplitudes of all motion vectors are counted, the amplitude mean value and the amplitude variance of the motion vectors in the background analysis region are calculated, and the background dynamic interference intensity of each frame of video in the background analysis region is determined according to the amplitude mean value and the amplitude variance; The trajectory point credibility of each time point in the double-ankle Y-coordinate trajectory data is evaluated according to the background region change stability coefficient and the background dynamic interference intensity. The initial credibility values of the trajectory points of the continuous frame sequence are smoothed based on a hidden Markov model, the credibility of the trajectory point at each time point is taken as an observation state of the hidden Markov model, and the credibility sequence of the trajectory points is obtained by decoding through a Viterbi algorithm; The Y-coordinate trajectory data is filtered based on a Gaussian low-pass filter, the filtering strength of each trajectory point in the Y-coordinate trajectory data is determined according to the credibility sequence of the trajectory points, and the filtering parameters of the Gaussian low-pass filter are determined according to the filtering strength, the filtering parameters including a filter kernel size and a standard deviation; The Y-coordinate trajectory data is filtered based on the filtering parameters to obtain a Y-trajectory sequence of the double-ankle movement. 4.The method of claim 1, wherein, The key audio points are identified according to the speech spectrum feature vectors, the key audio points including a start audio point, a foot landing audio point and an end audio point, and specifically comprising the following steps: Historical skipping rope audio data is acquired, speech frequency domain feature vectors of the historical skipping rope audio data are extracted, a key audio point acquisition time period of the historical skipping rope audio data is obtained, and historical speech frequency domain feature vectors of the key audio points are extracted according to the acquisition time period; The historical speech frequency domain feature vectors of each key audio point are labeled to obtain a training data set; A key audio point identification model is constructed based on a deep neural network, the model including an input layer, a hidden layer and an output layer, the key audio point identification model is trained using the training data set, and network weight parameters are optimized by minimizing a cross-entropy loss function through a back propagation algorithm; The real-time acquired speech spectrum feature vectors are input into the trained key audio point identification model to obtain a probability distribution of each audio frame key audio point; The key audio points in the speech spectrum feature vectors are identified according to the probability distribution, and key audio point type data is output.

5. The method of claim 1, wherein the method is a method of automatically counting a number of jumps of a user using a rope skipping device based on a visual and voice cooperative recognition. The number of effective jumps of the skipping rope user is calculated according to the effective jump wave peaks, and specifically comprising the following steps: Timestamp information corresponding to each effective jump wave peak is extracted, the effective jump wave peaks are time-sequenced according to the timestamp information, and an effective jump wave peak time sequence is constructed; An effective jump counter is initialized, a jump continuity determination time window is set, a time interval between adjacent effective jump wave peaks in the effective jump wave peak time sequence is calculated, and it is determined whether the time interval between the adjacent effective jump wave peaks is less than the jump continuity determination time window; When the time interval between the adjacent effective jump wave peaks is less than the jump continuity determination time window, it is determined that the adjacent effective jump wave peaks belong to the same continuous jump sequence, and the adjacent effective jump wave peaks are merged into the same continuous jump sequence; When the time interval between the adjacent effective jump wave peaks is not less than the jump continuity determination time window, it is determined that the current continuous jump sequence ends, and a new continuous jump sequence is recorded; The number of effective jump wave peaks in each continuous jump sequence is calculated, and the number of effective jump wave peaks in all effective jump sequences is accumulated to obtain the total number of effective jumps of the skipping rope user.

6. A skipping rope automatic counting system based on visual and voice collaborative recognition, characterized in that, The skip rope automatic counting system based on visual and voice cooperative recognition comprises a storage and a processor, the storage comprises a skip rope automatic counting method program based on visual and voice cooperative recognition, and the skip rope automatic counting method program based on visual and voice cooperative recognition is executed by the processor to realize the following steps: A skip rope video stream and a synchronously collected audio stream of a skipper are acquired, human body key point detection is performed on the video stream, Y coordinate track data of double ankle in continuous frames is extracted, and short-time Fourier transform is performed on the audio stream to generate a voice spectrum feature vector; Background interference recognition is performed on the skip rope video stream, the track point credibility of each time point in the Y coordinate track data is determined according to the background interference recognition, Gaussian low-pass filter noise processing is performed on the Y coordinate track data according to the track point credibility, and a double ankle movement Y track sequence of the skipper is formed; Key audio points are recognized according to the voice spectrum feature vector, the key audio points include a start audio point, a foot landing audio point and an end audio point; A double foot gait cycle of the skipper is recognized according to the double ankle movement Y track sequence, time alignment operation is performed on the key audio points and the double foot gait cycle, and effective jump wave peaks of the double ankle movement Y track sequence are recognized, specifically as follows: Local maximum value and local minimum value detection is performed on the double ankle movement Y track sequence based on a sliding time window, adjacent maximum value and minimum value are combined to form candidate wave peaks and candidate wave troughs, the amplitude difference between continuous candidate wave peaks and candidate wave troughs is calculated as a jump amplitude, and the time interval between continuous candidate wave peaks is calculated as a jump cycle; A preliminary double foot gait cycle sequence is constructed according to the jump amplitude and the jump cycle, a gait cycle segment located after the start audio point and before the end audio point is intercepted from the double foot gait cycle sequence as an analysis cycle segment according to the recognized start audio point and end audio point; Each foot landing audio point in the analysis cycle segment is matched with the most adjacent candidate wave trough in time, if the matching is successful, the candidate wave trough is marked as a confirmed landing point, Y track subsequences in a preset time window before and after the confirmed landing point are extracted, and waveform features of the Y track subsequences are calculated, the waveform features include wave peak steepness, wave trough flatness and wave peak and wave trough track smoothness; Based on the waveform features corresponding to all the confirmed landing points, a normal jump waveform feature template is generated by using a clustering algorithm, the normal jump includes take-off and periodic cyclic jump; Similarity calculation is performed on Y track subsequences corresponding to all the candidate wave peaks in the analysis cycle segment and the normal jump waveform feature template, if the similarity is lower than a preset threshold, it is determined that the Y track subsequence corresponding to the candidate wave peak is an interference sequence generated by skip rope failure and is removed; Candidate wave peaks with similarity higher than the preset threshold are reserved and output as an effective jump wave peak sequence; The effective jump number of the skipper is calculated according to the effective jump wave peak.

Images