Multi-person rope skipping counting method based on target identification

Abstract

The invention provides a multi-person rope skipping counting method based on target recognition, and belongs to the technical field of image processing of target recognition, and the method comprises the steps: S1, multi-source data collection and synchronization; s2, image preprocessing and target identification; s3, carrying out attitude processing and track fitting; s4, voice semantic analysis and strategy generation; s5, carrying out LSTM time sequence analysis and probability calculation; s6, multi-source information fusion counting decision making; and S7, result visualization. According to the method, YOLOv5 + LSTM + curved surface fitting combination is adopted, so that the computing power requirement is greatly reduced, and the hardware cost of an edge end is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing for target recognition, and in particular to a method for counting multiple people's rope skipping based on target recognition. Background Art

[0002] Automatic rope skipping counting is currently widely used in the field of smart sports and health, suitable for scenarios such as sports competitions, daily training, and physical education exams. It is generally divided into two types: real-time rope skipping counting and post-processing rope skipping counting. Compared with post-processing rope skipping counting, real-time rope skipping counting provides users with more immediate feedback and better dynamic interaction. It also has higher user satisfaction than post-processing rope skipping counting and is currently a goal that the industry is striving to achieve.

[0003] However, the existing real-time rope skipping counting solutions on the market have certain defects:

[0004] Sensor type: Currently widely used in the physical education exams for the National College Entrance Examination and the National High School Entrance Examination. The principle is to collect the user's motion data through photoelectric sensors and pressure sensors, and the motion data is transmitted to the terminal via wires. The terminal then calculates and counts the number of jumps in real time. However, the external sensor needs to be installed on the ground and connected to the terminal via a cable. This method increases the size and weight of the device and has poor flexibility. Under daily training conditions, this bloated method can easily damage the equipment, while limiting the installation location and may also cause cable wear and accidental tripping. Moreover, a set of sensor-based real-time jump rope counting solutions can usually only serve one user, and it is difficult to cope with the simultaneous counting needs of multiple users during daily training (such as physical education classes).

[0005] Human body key point recognition: Currently, there are real-time rope skipping counting solutions on the market based on visual human body key point recognition. The principle is to calculate the user's human body key points through deep learning algorithms. The number of rope skipping is usually calculated by the up and down movement of the hip key point exceeding a set threshold. However, due to current technological limitations, the deep learning algorithm for human body key points requires high computing power, while the computing power provided by the edge is relatively weak. Under the premise that the inference speed can meet the requirements, only single-person real-time rope skipping counting can be achieved. Using cloud computing that can provide high computing power and then transmitting it to the terminal via the network requires users to pay additional cloud computing and network transmission fees. It is also not suitable for schools in remote mountainous areas.

[0006] Failure in multi-person scenarios: The existing edge computing power can only support real-time counting of a single person. When multiple people jump rope at the same time, the inference delay soars and the counting accuracy drops sharply.

[0007] Common problems of existing pure vision solutions: they do not combine motion physics modeling, rely only on single visual features (such as the hip key point), are easily affected by camera shake and light changes, and have a high rate of missing small jumps.

[0008] The system lacks the ability to understand scene semantics and cannot adaptively adjust counting strategies (for example, strict counting is required in exam mode, while loose counting is required in training mode), resulting in a poor user experience. Summary of the Invention

[0009] The present invention provides a multi-person rope skipping counting method based on target recognition to solve one or more of the above problems.

[0010] The multi-person rope skipping counting method based on target recognition includes:

[0011] S1. Multi-source data acquisition and synchronization: The camera collects color image signals, the microphone array collects voice command signals, and the gyroscope collects attitude signals. These signals are then timestamped and aligned to produce a multi-source signal with aligned timestamps.

[0012] S2. Image Preprocessing and Target Recognition: Perform bilinear interpolation scaling and normalization on the color image signals from the multi-source signals to obtain a preprocessed image. Use the YOLOv5 model to perform target recognition on the preprocessed image and obtain the human target coordinate signal.

[0013] S3. Posture Processing and Trajectory Fitting: Kalman filtering is performed on the posture signals from the multi-source signals. The terminal rotation angle increment is calculated by integration to generate a jitter compensation signal. Based on the human target coordinate signal and the jitter compensation signal, a surface fitting is performed using a sine function superimposed on linear terms to generate a smooth jump trajectory signal. The first-order derivative of the smooth jump trajectory signal is calculated to obtain a trajectory velocity derivative signal. The maximum value of the current cycle is extracted from the trajectory velocity derivative signal to obtain the trajectory velocity peak feature.

[0014] S4. Speech Semantic Parsing and Strategy Generation: This involves performing noise reduction and speech recognition on multi-source speech command signals to generate semantic text. A lightweight large language model is used to parse scene semantics and motion patterns. This is combined with the amplitude of the jitter compensation signal to generate a dynamic parameter adjustment signal containing a threshold adjustment factor and speed weight.

[0015] S5. LSTM Timing Analysis and Probability Calculation: The smooth jump trajectory signal and trajectory velocity derivative signal are input into the LSTM network. The forget gate, input gate, and output gate mechanism are used to analyze the timing characteristics and output the jump probability signal.

[0016] S6. Multi-source information fusion counting decision: The trajectory velocity peak feature is weighted according to the velocity weight in the dynamic parameter adjustment signal. The weighted trajectory velocity peak feature is multiplied by the jump probability signal. The result is then compared with the dynamic threshold calculated from the threshold adjustment factor in the dynamic parameter adjustment signal to generate a counting decision signal. When the decision condition is met, the corresponding area counter is incremented by 1.

[0017] S7. Result visualization: The counting decision signal is superimposed on the original image to generate a visual counting result.

[0018] In this specification, the color image captured by the camera in S1 is an RGB signal with a resolution of 1920×1080. When it is scaled to 608×608 pixels through bilinear interpolation, the weighted average formula is used to calculate the new pixel value. After normalization, the pixel value range is [-1, 1].

[0019] In this specification, the microphone array in the S1 is configured with an 8cm spacing, achieves ±3° sound source positioning through a phase difference algorithm, and attenuates 90dB of ambient noise in combination with a dynamic noise suppression filter.

[0020] In this specification, the processing of the attitude signal in S3 includes: fusing the three-axis angular velocity data of the previous 100ms through the Kalman filter, using the trapezoidal integration method to calculate the terminal rotation angle increment, and the generated jitter compensation signal is used to correct the linear drift term of the surface fitting.

[0021] In this specification, the training data of the YOLOv5 model in S2 contains 20,000 images of multiple people skipping rope, covering scenes of 1 to 50 people, users aged 7 to 50, and sunny / cloudy / indoor lighting conditions. Mosaic data enhancement and cosine annealing learning rate strategy are used to optimize the model until the average precision calculated is greater than or equal to 0.96 when the intersection-over-union ratio threshold is 0.5.

[0022] In this specification, the mathematical model used for surface fitting in S3 is: , add regularization terms when solving parameters by least squares method , the RMS fitting error is less than 4 pixels, where For the Trajectory fitting function of the region, is the jump amplitude, which represents the user's jump height; is the angular frequency, , is the hopping frequency; is the phase shift, which represents the difference in jump starting posture; is the linear drift coefficient, compensating for the user's forward and backward movement; is the vertical reference, corresponding to the reference coordinate when the user is standing; is Gaussian noise, is the time variable, is the regularization coefficient.

[0023] In this specification, the lightweight large language model in S4 is built based on the LLaMA-7B architecture, and the input layer receives the amplitude of the jump trajectory , angular frequency , linear drift coefficient and semantic text embedding vectors, the threshold adjustment factor generated by the output layer ranges from [-0.3, 0.3], and the speed weight range is [0, 1].

[0024] In this specification, the generation rule of the counting decision signal in S6 is: when the jump probability × (1 + speed weight × speed peak / 30) > dynamic threshold, it is determined to be a valid jump, where:

[0025] The velocity peak is the maximum value of the trajectory velocity derivative signal in the current cycle;

[0026] The dynamic threshold is calculated by the basic threshold × (1 + threshold adjustment factor), and the basic threshold is preset to 0.6.

[0027] In this specification, the specific implementation of the surface fitting in S3 includes:

[0028] Input 30 consecutive frames of human target coordinate signals into the Kalman filter and output a smooth coordinate sequence; perform fast Fourier transform on the smooth coordinate sequence and extract the main frequency component as the initial frequency parameter estimated value of;

[0029] The parameters of the sinusoidal model are optimized by the nonlinear least squares method, and the iterative convergence condition is that the parameter change rate between two adjacent iterations is less than 0.01%.

[0030] In this specification, the generation rule of the counting decision signal in S6 further includes:

[0031] When the jump probability of three consecutive cycles is greater than 0.9, the fast confirmation mechanism is triggered and it is directly determined to be a valid jump;

[0032] When the peak speed / reference speed is detected to be greater than 1.5, the double integration mode is activated and the corresponding area counter is incremented by 2;

[0033] When the amplitude of the jitter compensation signal is greater than 0.8 rad / s, the anti-interference enhancement mode is activated and the dynamic threshold is increased by 50%.

[0034] The embodiments of this specification can achieve at least the following beneficial effects:

[0035] Low computing power and efficient counting: The YOLOv5+LSTM+surface fitting combination significantly reduces computing power requirements and lowers edge hardware costs.

[0036] Energy consumption optimization: The overall power consumption is low (compared to traditional sensor solutions), and it can be powered by batteries, making it suitable for outdoor scenarios without power supply (such as playgrounds and schools in mountainous areas).

[0037] Multi-user concurrency: A single device supports counting multiple users simultaneously (determined by the camera's field of view). The number of users has little impact on system performance, and counting efficiency is improved in physical education scenarios.

[0038] Accuracy improvement details: Through surface fitting to filter noise and LLM dynamic threshold adjustment, the missed count rate of small jumps is reduced, and the miscount rate in occluded scenes is reduced.

[0039] Multi-mode automatic switching: LLM can identify scenarios such as "exam", "training", and "children", and automatically adjust thresholds and counting strategies.

[0040] Natural language interaction: supports voice commands (such as "start counting" and "switch to high school entrance examination mode"), with low response latency, and is suitable for visually impaired users or scenarios where multiple people are operating simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 Schematic diagram of a method for counting multiple people skipping rope based on target recognition involved in some embodiments of the present invention.

[0043] Figure 2 Schematic diagram of the hardware system involved in some embodiments of the present invention.

[0044] Figure 3 Schematic diagram of the software system involved in some embodiments of the present invention.

[0045] Figure 4 Schematic diagram of screen display involved in some embodiments of the present invention.

[0046] Figure 5 Schematic diagram of the structure of the counting module processing flow involved in some embodiments of the present invention.

[0047] Figure 6 Schematic diagram of the YOLOv5 network structure involved in some embodiments of the present invention.

[0048] Figure 7 Schematic diagram of the Y-axis coordinates, probability output, and decision threshold of the original user and the pre-processed user involved in some embodiments of the present invention.

[0049] Figure 8 Schematic diagram of the LSTM time series processing network involved in some embodiments of the present invention. DETAILED DESCRIPTION

[0050] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and description are to be considered as illustrative in nature and not restrictive.

[0051] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the products of the present invention are conventionally placed when in use, or are the orientations or positional relationships conventionally understood by those skilled in the art. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as limiting the present invention.

[0052] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0053] Furthermore, the terms "mounted," "connected," "connect," and "fixed" should be interpreted broadly. For example, they may refer to fixed connection, detachable connection, or integration; they may refer to direct connection or indirect connection through an intermediate medium; they may refer to internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0054] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0055] like Figure 1 As shown, this embodiment provides a method for counting multiple people skipping ropes based on target recognition, including:

[0056] S1. Multi-source data acquisition and synchronization: Camera 2 collects color image signals, a microphone array collects voice command signals, and a gyroscope collects attitude signals. These signals are then timestamped and aligned to produce a timestamp-aligned multi-source signal.

[0057] S2. Image Preprocessing and Target Recognition: Perform bilinear interpolation scaling and normalization on the color image signals from the multi-source signals to obtain a preprocessed image. Use the YOLOv5 model to perform target recognition on the preprocessed image and obtain the human target coordinate signal.

[0058] S3. Posture Processing and Trajectory Fitting: Kalman filtering is performed on the posture signals from the multi-source signals. The rotation angle increment of terminal 1 is calculated by integration to generate a jitter compensation signal. Based on the human target coordinate signal and the jitter compensation signal, a surface fitting is performed using a sine function superimposed on linear terms to generate a smooth jump trajectory signal. The first-order derivative of the smooth jump trajectory signal is calculated to obtain a trajectory velocity derivative signal. The maximum value of the current cycle is extracted from the trajectory velocity derivative signal to obtain the trajectory velocity peak feature.

[0059] S4. Speech Semantic Parsing and Strategy Generation: This involves performing noise reduction and speech recognition on multi-source speech command signals to generate semantic text. A lightweight large language model is used to parse scene semantics and motion patterns. This is combined with the amplitude of the jitter compensation signal to generate a dynamic parameter adjustment signal containing a threshold adjustment factor and speed weight.

[0060] S5. LSTM Timing Analysis and Probability Calculation: The smooth jump trajectory signal and trajectory velocity derivative signal are input into the LSTM network. The forget gate, input gate, and output gate mechanism are used to analyze the timing characteristics and output the jump probability signal.

[0061] S6. Multi-source information fusion counting decision: The trajectory velocity peak feature is weighted according to the velocity weight in the dynamic parameter adjustment signal. The weighted trajectory velocity peak feature is multiplied by the jump probability signal. The result is then compared with the dynamic threshold calculated from the threshold adjustment factor in the dynamic parameter adjustment signal to generate a counting decision signal. When the decision condition is met, the corresponding area counter is incremented by 1.

[0062] S7. Result visualization: The counting decision signal is superimposed on the original image to generate a visual counting result.

[0063] In some embodiments, the color image captured by the camera 2 in S1 is an RGB signal with a resolution of 1920×1080. When it is scaled to 608×608 pixels through bilinear interpolation, a weighted average formula is used to calculate the new pixel value, and the pixel value range after normalization is [-1,1].

[0064] In some embodiments, the microphone array in S1 is configured with an 8 cm spacing, achieves ±3° sound source positioning through a phase difference algorithm, and is combined with a dynamic noise suppression filter to attenuate 90 dB of ambient noise.

[0065] In some embodiments, the processing of the attitude signal in S3 includes: fusing the three-axis angular velocity data of the previous 100ms through a Kalman filter, using a trapezoidal integration method to calculate the rotation angle increment of the terminal 1, and generating a jitter compensation signal for correcting the linear drift term of the surface fitting.

[0066] In some embodiments, the training data of the YOLOv5 model in S2 includes 20,000 images of multiple people skipping rope, covering scenes of 1 to 50 people, users aged 7 to 50, and sunny / cloudy / indoor lighting conditions. Mosaic data enhancement and cosine annealing learning rate strategy are used to optimize the model until the average precision calculated is greater than or equal to 0.96 when the intersection-over-union ratio threshold is 0.5.

[0067] In some embodiments, the mathematical model used for surface fitting in S3 is: , add regularization terms when solving parameters by least squares method , the RMS fitting error is less than 4 pixels, where For the Trajectory fitting function of the region, is the jump amplitude, which represents the user's jump height; is the angular frequency, , is the hopping frequency; is the phase shift, which represents the difference in jump starting posture; is the linear drift coefficient, compensating for the user's forward and backward movement; is the vertical reference, corresponding to the reference coordinate when the user is standing; is Gaussian noise, is the time variable, is the regularization coefficient.

[0068] In some embodiments, the lightweight large language model in S4 is constructed based on the LLaMA-7B architecture, and the input layer receives the amplitude of the jump trajectory , angular frequency , linear drift coefficient and semantic text embedding vectors, the threshold adjustment factor generated by the output layer ranges from [-0.3, 0.3], and the speed weight range is [0, 1].

[0069] In some embodiments, the counting decision signal generation rule in S6 is: when the jump probability × (1 + speed weight × speed peak / 30) > dynamic threshold, it is determined to be a valid jump, where:

[0070] The velocity peak is the maximum value of the trajectory velocity derivative signal in the current cycle;

[0071] The dynamic threshold is calculated by the basic threshold × (1 + threshold adjustment factor), and the basic threshold is preset to 0.6.

[0072] In some embodiments, the specific implementation of the surface fitting in S3 includes:

[0073] Input 30 consecutive frames of human target coordinate signals into the Kalman filter and output a smooth coordinate sequence; perform fast Fourier transform on the smooth coordinate sequence and extract the main frequency component as the initial frequency parameter estimated value of;

[0074] The parameters of the sinusoidal model are optimized by the nonlinear least squares method, and the iterative convergence condition is that the parameter change rate between two adjacent iterations is less than 0.01%.

[0075] In some embodiments, the generation rule of the counting decision signal in S6 further includes:

[0076] When the jump probability of three consecutive cycles is greater than 0.9, the fast confirmation mechanism is triggered and it is directly determined to be a valid jump;

[0077] When the peak speed / reference speed is detected to be greater than 1.5, the double integration mode is activated and the corresponding area counter is incremented by 2;

[0078] When the amplitude of the jitter compensation signal is greater than 0.8 rad / s, the anti-interference enhancement mode is activated and the dynamic threshold is increased by 50%.

[0079] The technical concept of the present invention is as follows:

[0080] A valid jump is determined based on the Y-axis (vertical) change characteristics of the user identified by the object recognition neural network (YOLOv5) using a long short-term memory neural network (LSTM). The system primarily consists of hardware and software systems. The maximum number of people counted simultaneously depends on the field of view of camera 2. A larger field of view allows for more users to be simultaneously counted, resulting in a greater maximum number of people being counted simultaneously. The number of people has minimal impact on system performance.

[0081] Hardware system such as Figure 2 As shown, Figure 2 middle:

[0082] Terminal 1 is made of stainless steel sheet metal and houses the power supply module, computing module, and other components. A screen 3 and camera 2 for user interaction are located on the front, and the user stands directly in front of terminal 1.

[0083] 2 is a camera: installed in the center of the upper front of the terminal 1, used to collect visual information.

[0084] 3 is a screen: installed in the center of the front of the terminal 1, used for interaction with the user.

[0085] The software system is based on the Android framework and consists of two parts (such as Figure 3 As shown). Among them, the front-end part is mainly used for interaction with the user. The camera image module obtains image data from the camera 2 and transmits the image data to the screen display module and the counting module; the screen display module superimposes the image data from the camera image module and the counting result from the counting module and displays them on the screen 3 (as shown). Figure 4 As shown in the figure): The top of screen 3 is a countdown display, which is composed of multiple recognition areas. When there is a user in the recognition area, the cumulative number of jumps by the user will be displayed. When there is no user in the recognition area, the recognition area will display "No user recognized". The counting module in the software system is the core part, and the processing flow (such as Figure 5 shown).

[0086] Detailed introduction to the real-time counting process (Example 1):

[0087] 1. Obtaining Raw Image Data: The software system's camera imaging module uses the hardware system's camera 2 via the USB-UVC protocol to acquire a color, three-channel image with a resolution of 1920 × 1080 at 30 frames per second. This image is then transmitted to the counting module and the screen display module. This provides the raw image for image preprocessing.

[0088] Image preprocessing: The original 1920×1080 three-channel image is scaled to a 608×608 three-channel image using linear interpolation. The resized image is then normalized, mapping pixel values from [0, 255] to [-1, 1]. The preprocessed image is then fed into the object recognition model for processing.

[0089] 2. YOLOv5 target recognition: YOLOv5 algorithm is a target detection algorithm based on deep learning (such as Figure 6 The core idea is to treat target detection as a regression problem and directly predict the location and category of the target through a convolutional neural network. The network structure of YOLOv5 is as follows Figure 6 As shown, Figure 6In the network, Focus: focus (extract features from the input image and downsample), CBL: convolution layer-batch normalization-activation function, CSPx_x: cross-stage part, SPP: spatial pyramid pooling, Concat: splicing, conv: convolution, BN: batch normalization, Leaky relu: leaky rectified linear unit (an activation function), resunit: residual unit, slice: slice. The network structure of YOLOv5 mainly includes four modules: Input, Backbone backbone network, Neck structure and Head structure (Prediction), which are responsible for input image preprocessing, feature extraction, feature fusion, and output detection information respectively. Among them, the Backbone module uses the BottleNeckCSP structure to extract rich information features from the input image. After the 608×608×3 preprocessed image is input into the YOLOv5 target recognition model, the model output is a 6×N two-dimensional array with the following structure:

[0090] ;

[0091] The size N of the two-dimensional array represents the number of identified targets, and each row represents a set of attributes of a target. Representative Target The position of the center point on the X-axis coordinate axis of the input image, Representative Target The position of the center point on the Y-axis of the input image, Representative Target The width of the recognition box, Representative Target The height of the recognition frame, Representative Target The category of is only 0 in this invention, representing the "human" category, Representative Target The confidence level is the probability value of the target recognition result being true.

[0092] At this point, the original coordinate information of all "human" targets within the camera's field of view is obtained, so as to determine which ones are real rope skipping users.

[0093] 3. Determine whether there are users in the recognition area: After obtaining the original coordinate information of all "people" in the camera's field of view, it is necessary to filter them to determine whether they are real rope jumpers in the detection area, because the original recognition results contain false recognition caused by noise and non-rope jumpers (onlookers, etc.) in the non-detection area. The vector function of the filtering condition is:

[0094] ;

[0095] in, Representative about the goal Is it a region The Boolean function of a real rope skipping user, the symbol "^" represents the logical AND operation, Representative Target The confidence level, Representative Target The position of the center point on the X-axis coordinate axis of the input image, Representative Target The position of the center point on the Y-axis of the input image, 、 、 、 Representative area The coordinate range of the valid rectangular area. This function can be used to filter out low-confidence noise misidentifications and non-rope skipping users who are not in the valid detection frame.

[0096] The result of the vector function is actually a 1×m one-hot code. When the result of a row is true, it means that the target recognition result is a real rope skipping user in the detection area. Input to the next module; when all rows of the one-hot code are false, zero value (0) is input to the next module.

[0097] At this point, the real rope skipping users in the specified recognition area are filtered out from all the target recognition results about "people", and the original user Y-axis coordinates in each recognition area are obtained. If there is no user in the recognition area, the original user Y-axis coordinate is zero (0).

[0098] 4. Data preprocessing: The time domain image of each original user's Y-axis coordinate with respect to time is similar to a periodic function. However, due to the different front and back positions of different user stations, and the result of the target recognition algorithm is noisy (such as Figure 7 The original user Y-axis coordinates are shown in the figure), and preprocessing is required. The preprocessing formula is as follows:

[0099] ;

[0100] in, Representative time point Original user Y-axis coordinate, Representative time point User Y-axis coordinate after preprocessing, represents the size parameter of the preprocessing low-pass filter, represents the preprocessing gain parameter, Represents the preprocessing bias parameter.

[0101] So far, we have obtained the pre-processed user Y-axis coordinates (such as Figure 7 The waveform and first-order derivative of this data are relatively smooth, which can be used by the LSTM time series processing network to determine whether the user has performed a valid rope skipping action.

[0102] 5. LSTM Time Series Processing: LSTM time series processing network is a kind of RNN-like time series processing network. The output at this moment depends not only on the input at this moment, but also on the hidden input at this moment (the hidden output at the previous moment). It has a memory-like function. Based on this principle, it can be used to identify whether a complete waveform has appeared. LSTM time series processing network (such as Figure 8 As shown in Figure 2, the LSTM time series processing network consists of multiple LSTM modules and fully connected layers. The processing flow of each LSTM module in the LSTM time series processing network is as follows:

[0103] First, decide which part of the hidden layer output at the previous moment needs to be forgotten:

[0104] ;

[0105] in, and represents the weight, and Represent the hidden part of the hidden output at the previous moment and the input at the current moment, represents the output of the forget gate, Represents vector concatenation operation, represents the S-type activation function, which is:

[0106] ;

[0107] Next, decide which part of the input at this moment needs to be added to the cell part of the hidden output at this moment:

[0108] ;

[0109] in, 、 、 、 Represent the corresponding weights, 、 Represent the vectors of the input gate output and the new candidate value respectively, represents the hyperbolic tangent function, which is:

[0110] ;

[0111] Next, update the cell state:

[0112] ;

[0113] in, Represents the new cell state, which is obtained by forgetting part of the old cell state through the forget gate and adding part of the new candidate value through the input gate:

[0114] Finally, the calculation determines the final output at this moment through the cell state:

[0115] ;

[0116] in, 、 Represent the corresponding weights, Represents the output of the output gate, which determines which parts of the cell state can be output. Represents the output of the final LSTM module.

[0117] After the above processing, we can obtain the output of multiple LSTM modules, which can be flattened into a vector A. The formula is as follows:

[0118] ;

[0119] in, Representative The output of the LSTM module at the current moment, is the number of LSTM modules.

[0120] Finally, calculate the final probability output:

[0121] ;

[0122] in, represents the weight, Represents the probability that the user has completed a valid jump at the current moment, ranging from [0,1].

[0123] So far, we have obtained the probability that the user has completed a valid jump at each time moment (e.g. Figure 7 ) for counting rope skipping.

[0124] 6. Probability judgment and counting: Each processing is to judge the probability output. When the probability output is greater than the threshold (such as Figure 7 The counter will be incremented by 1.

[0125] At this point, the number of rope skipping by users in each detection area is obtained.

[0126] 7. Screen 3 display: Through the screen display module of the software system, the original image data and the counting results are superimposed, and the number of rope skipping by users in the detection area is displayed on screen 3 in real time.

[0127] At this point, real-time rope skipping counting for multiple people is achieved.

[0128] Based on the above solution, the optimal solution of the present invention is obtained by combining surface fitting and large language model, which is as follows (Example 2):

[0129] 1. Hardware system upgrade and cross-module collaborative architecture

[0130] 1.1 Integrated support design of hardware modules

[0131] The voice interaction module is deeply integrated: It utilizes a dual-microphone array (8cm apart) combined with beamforming technology. This technology calculates the direction of the sound source using phase difference (with an accuracy of ±5°), enabling real-time filtering of non-user speech (such as background noise). The audio processing chip (WM8960) features a built-in noise suppression algorithm that attenuates ambient noise (such as playground cheers) by 90dB, achieving a 92% accuracy rate in voice command recognition. The processed text sequence (e.g., "Middle School Exam Mode") is transmitted to the edge computing unit via shared memory with a transmission latency of ≤80ms.

[0132] Heterogeneous computing architecture optimization: The RV1126 chip (1.5TOPS) and the RV1106G3 chip (0.5TOPS) form a heterogeneous computing cluster via the PCIe 2.0 bus (4GB / s bandwidth). The RV1126 is dedicated to large language model inference (accounting for 60% of the computing power), while the RV1106G3 is responsible for visual processing (40% of the computing power). Resources are dynamically allocated through the task scheduler. For example, if more than five people are detected jumping rope, the RV1126 automatically increases the LLM inference computing power to 80%, ensuring semantic decision latency ≤ 100ms.

[0133] Gyroscope dynamic calibration module: The MPU6050 gyroscope samples three-axis angular velocity at 100Hz (resolution 0.01 rad / s) and fuses historical attitude data through a Kalman filter to generate a real-time jitter compensation matrix. When the detected X-axis angular velocity exceeds 0.2 rad / s, an offset Δy = 15 ∫ωxdt (in pixels) is added to the Y-axis coordinate output by YOLOv5 to eliminate coordinate errors caused by camera 2 tilt (compensation accuracy ≤ 3 pixels).

[0134] 1.2 Cross-hardware data synchronization mechanism

[0135] A nanosecond-level timestamp synchronization system was established: All modules (Camera 2, microphone, and gyroscope) generate timestamps using the main control unit's crystal oscillator (accuracy ±1ppm). The timestamp error for visual frames, voice commands, and gesture data is ≤10μs. For example, the image captured by Camera 2 at t=10.250s is precisely aligned with the gyroscope sampled at t=10.2501s, ensuring accurate motion compensation during trajectory fitting.

[0136] 2. Image Acquisition and Preprocessing (Compatible with the Original Algorithm of Example 1)

[0137] 2.1 The HD image acquisition process uses a 1080P global shutter camera, capturing RGB images at a frame rate of 30 fps, with a single frame size of 6MB (uncompressed). YUY2 encoding is used for transmission via the USB 3.0 interface, preserving color information while reducing data size by 30% (transmission bandwidth ≤ 120MB / s), keeping the frame loss rate below 0.1%. To accommodate subsequent processing, the image sensor's automatic gain control (AGC) and automatic white balance (AWB) parameters are dynamically adjusted by software to ensure image contrast ≥ 300:1 in scenes such as overcast skies (1000 lux) and indoor lighting (500 lux).

[0138] 2.2 Preprocessing algorithm optimization details

[0139] Bilinear interpolation scaling: For the target pixel, calculate its mapping coordinates in the original image and generate a new pixel value by weighted averaging the surrounding 4 pixels.

[0140] This method reduces the jagged edges of scaled human figures by 40% and improves YOLOv5's recognition rate for small objects by 5%. Normalization: Mapping pixel values from [0, 255] to [-1, 1];

[0141] The variance of the processed data is controlled between 0.5 and 0.8, ensuring a stable input distribution of the convolutional layer of YOLOv5 and increasing the convergence speed by 20%.

[0142] 3. YOLOv5 target recognition model processing

[0143] 3.1 Model Architecture and Output Definition

[0144] Using YOLOv5s-v6.2 version, the network structure includes:

[0145] Input module: Mosaic data enhancement (4-image stitching) + adaptive anchor box calculation to improve small object detection capabilities in multi-person scenes;

[0146] Backbone module: CSPDarknet-53, which reduces computational complexity by 30% through cross-stage partial connections while retaining 90% of feature extraction capabilities;

[0147] Neck module: FPN+PAN bidirectional feature fusion achieves three scale detections (19×19 / 38×38 / 76×76), with a recognition rate of ≥85% for long-distance users (>5 meters);

[0148] Head module: Outputs a 6×N array, where N is the number of targets and each row contains:

[0149] : Target center coordinates (pixel value, accuracy ±2);

[0150] : Width and height of target frame (used to judge the integrity of human body);

[0151] : Confidence (0-1, representing the probability of being a human).

[0152] 3.2 Model training process

[0153] Dataset Construction: 20,000 images of people skipping rope were collected, covering scenes with 1 to 50 people, users aged 7 to 50, and sunny, cloudy, and indoor lighting conditions. LabelImg was used to annotate the human body bounding boxes (with an error of < 5 pixels), and the training / validation / test sets were divided into 8:1:1 ratios.

[0154] Training strategy: Transfer learning: Based on COCO pre-trained weight initialization Optimizer: SGD (momentum 0.937, weight decay 5e-4), the learning rate decays from 0.01 to 0.0001 through cosine annealing Data augmentation: random flip, rotation (±15°), HSV adjustment (hue ±10, saturation ±15%, lightness ±15%) Model pruning: Remove redundant connections after 50 rounds of training. The final model size is 12MB (INT8 quantization), and the edge inference latency is ≤15ms.

[0155] 3.3 Target filtering and coordinate extraction

[0156] Triple filtering mechanism:

[0157] Confidence filtering: ≥0.7, excluding background misidentification;

[0158] Area Filter: , , preset 3 recognition areas (left / center / right); : X-axis range of the mth region, : Y-axis range of the mth region;

[0159] Size Filter: ≥5000 pixels², excluding distant tiny objects.

[0160] Generate a Y-axis coordinate sequence for each valid area m (30 frames / second) as input for surface fitting.

[0161] 4. Construction, training and application of surface fitting models

[0162] 4.1 Physical modeling and parameter definition

[0163] Jump trajectory fitting function:

[0164] Physical meaning of parameters:

[0165] : jump amplitude (standard jump for adults ≈ 40 pixels, for teenagers ≈ 25 pixels); : angular frequency, is the jumping frequency (professional athletes can reach 3Hz); : Phase offset (left-handed lags behind right-handed lag by π / 4); : Linear drift coefficient (the user moves forward 0.5m / s corresponding to ≈10 pixels / frame) : Vertical reference (Y-axis coordinate when the user is standing); : Gaussian noise ( ∼N(0, ²), the noise standard deviation of normal scenes =5 pixels.

[0166] 4.2 Model training implementation details

[0167] Multi-dimensional dataset collection:

[0168] Age stratification: 7-18 years old (300 people), 18-50 years old (200 people), with age group labels;

[0169] Action types: 6 modes including standard jump, high-leg jump, double-swing jump, etc.

[0170] Environmental variables: sunny day (5000 lux), cloudy day (1000 lux), indoor (500 lux).

[0171] Labeling process: Use the Vicon motion capture system (0.1mm accuracy) to obtain the real trajectory, align it with the data from Camera 2, and manually label the peak points of each jump cycle (error <5ms), generating 500,000 sets of ground-truth samples.

[0172] Parameter solving process:

[0173] Initial value setting: , (preset frequency 2Hz), ;

[0174] Objective function: , iterated by the Levenberg-Marquardt algorithm; is the original Y-axis coordinate value of the m-th recognition area at time t (output by the YOLOv5 target recognition model, unprocessed pixel value);

[0175] Regularization term: , = 0.01 to prevent overfitting, and the final MSE is < 4 pixels².

[0176] 4.3 Real-time fitting and dynamic parameter adjustment

[0177] User adaptive mechanism:

[0178] Novice detection: 5 consecutive cycles <20 and <1.5Hz, judged as "Novice Mode", Increased from 5 to 8 pixels to avoid small jumps and missed counts;

[0179] Fatigue compensation: A decrease of >30% for 10 consecutive seconds and If the fluctuation is >20%, reduce the LSTM threshold by 15%.

[0180] Gyro linkage correction:

[0181] Angular velocity vector detected by the gyroscope during shaking Norm of , When the pixel value increases to 10, the fitting function switches to the "anti-shake mode", or renew ;

[0182] Coordinate compensation: , eliminating the tilt error of camera 2.

[0183] Indicates the Y-axis angular velocity collected by the gyroscope Integrate over time from 0 to t. Where, Represents at the moment The angular velocity value of the gyroscope Y axis at this time, the role of integration is to accumulate the change in the angular velocity of the Y axis during this period of time to reflect the possible impact of factors such as the jitter of terminal 1 on the coordinates, and then multiply it by the calibration coefficient 0.1 and add it to On, realize compensation correction.

[0184] 5. Construction, Training, and Semantic Decision-Making of Large Language Models

[0185] 5.1 Lightweight LLM Architecture Design A dedicated model is built based on LLaMA-7B, with the following optimizations:

[0186] Structural compression: The number of Transformer layers was reduced from 32 to 6, and the hidden layer dimension was reduced from 4096 to 512, retaining 90% of the semantic understanding capability through knowledge distillation;

[0187] Input Design:

[0188] Visual features: (3D);

[0189] Semantic features: instruction embedding after BPE word segmentation (128 dimensions, vocabulary size 5000);

[0190] Output design:

[0191] Threshold adjustment factor ∈[-0.3,0.3];

[0192] Speed Weight ∈[0,1], normalized by sigmoid function.

[0193] 5.2 Cross-modal training process

[0194] Training data preparation:

[0195] Trajectory-strategy pairs: 10,000 sets of different action patterns (such as "small jump" and "fast jump") , mark the optimal parameters (such as "small jump" requires =-0.2);

[0196] Command-strategy pairs: 20 scenarios ("high school entrance exam", "children", etc.), each with 5-10 parameter templates (e.g. "high school entrance exam" =+0.2);

[0197] Abnormal action library: trajectory characteristics and corresponding strategies for 8 illegal actions (such as "tiptoe jumping") (threshold increased by 20%);

[0198] Training parameters: loss function: MSE (parameter regression) + cross entropy (scene classification);

[0199] Optimizer: AdamW (β1=0.9,β2=0.999), learning rate 1e-4;

[0200] Inference latency: <30ms per sample (RV1126, INT8 quantization).

[0201] 5.3 Three-level real-time decision-making process

[0202] Level 1: Scene semantic analysis

[0203] Voice commands use the BERT classification model to identify scenarios (such as "middle school entrance examination" and "training", with an accuracy rate of 95%). When there are no commands, they are inferred based on historical counts (for example, continuous counts > 180 → "exam mode").

[0204] Level 2: Trajectory Characteristics Understanding

[0205] enter Recognize action patterns, such as: <20 and >2.5Hz→"fast small jump", generating =-0.2 to compensate for insufficient height.

[0206] Level 3: Dynamic Feedback Optimization

[0207] Missing count correction: If three consecutive valid jumps in a certain area are not counted, record the user characteristics (such as =22), generate exclusive =-0.15 until the end of skipping.

[0208] 6. The entire counting decision process of the three-module integration

[0209] 6.1 Spatiotemporal Alignment of Cross-Module Data

[0210] Timestamp synchronization protocol:

[0211] Visual data: Camera 2 hardware timestamp (μs level);

[0212] Voice data: audio processing timestamp ,and Error <10ms;

[0213] Trajectory data: surface fitting timestamps , through sliding window interpolation and Align.

[0214] 6.2 Application Examples of Fusion Criteria

[0215] Scenario 1: Physical Education Test for the Senior High School Entrance Examination

[0216] LLM strategy: Analyze the "Secondary School Entrance Examination Model" and generate =+0.2 (threshold from 0.6→0.72), =0.8;

[0217] Surface fitting: From 5 to 3 pixels, filter high-frequency noise;

[0218] Counting calculation: candidates =30, =20, LSTM output 0.75, 0.75×(1+0.8×20 / 30)=0.75×1.53=1.15>0.72→count+1.

[0219] Scene 2: Primary school physical education class

[0220] LLM strategy: 10 people skipping rope without instructions → "training mode", = -0.3 (threshold → 0.42), =0.3;

[0221] Surface fitting: The effective threshold is from 20 to 15 pixels;

[0222] Counting: Primary School Students =18, =12, LSTM output 0.6, 0.6×(1+0.3×12 / 30)=0.6×1.12=0.67>0.42→count+1.

[0223] 6.3 Abnormal Situation Handling Logic

[0224] User Occlusion: YOLOv5 Confidence <0.5, but the surface fitting trajectory is continuous (fluctuation <5 pixels);

[0225] LLM determines "temporary occlusion", LSTM predicts the output 0.8 based on the historical trajectory, and the count takes effect.

[0226] Cheating Detection:

[0227] 5 consecutive cycles <10 and <1Hz → "slow jump cheat";

[0228] LLM generation = +0.3 (threshold → 0.78), screen 3 is marked as "suspected cheating", which requires both LSTM output > 0.78 and >15.

[0229] 7. Continuous Evolution and Resource Scheduling of Converged Systems

[0230] 7.1 Incremental Learning Feedback Loop

[0231] User-specific models:

[0232] Stores the user's last 100 jumps ;

[0233] When used again, LLM will first search the historical strategy, such as user A often =22, automatically generated =-0.18.

[0234] Global strategy library iteration:

[0235] Daily aggregated data, K-means clustering of new movement patterns (such as "new double shake jump");

[0236] Actively learn and label new categories, and release strategy library upgrade packages every quarter.

[0237] 7.2 Dynamic Computing Power Scheduling Strategy

[0238] Adaptive inference mode (some examples):

[0239] Number of people LLM layers Fitting sampling rate Delay Accuracy 1-2 6 30Hz <80ms 98.5% 3-5 4 25Hz <100ms 97.2% >5 2 20Hz <120ms 95.8%

[0240] Accuracy compensation: In lightweight mode, "historical trajectory prediction" is enabled, and the current frame is extrapolated from the previous 10 seconds, with an accuracy drop of ≤3%.

[0241] Three-module fusion technology map:

[0242] Data fusion chain: Camera 2 (image) → YOLOv5 (coordinates) → surface fitting (trajectory) → LSTM / LLM (decision) → count;

[0243] This chain realizes a cross-layer mapping from physical motion to semantic understanding, where surface fitting serves as a bridge to transform 2D visual coordinates into physiologically meaningful motion parameters, providing understandable semantic features for LLM.

[0244] Parameter interaction loop: (Adjustment) → Counting criterion ← Surface fitting / LSTM (error).

[0245] Evolutionary closed loop: counting results (analysis) → LLM strategy library (update) → subsequent counting.

[0246] The counting results will inversely affect the model parameters, forming a closed loop of "decision-feedback-optimization", giving the system the ability to self-evolve. For example, missed count feedback will prompt the LLM to adjust the threshold strategy, while jitter feedback will optimize the noise parameters of surface fitting.

[0247] In summary, multiple specific embodiments of the present invention are disclosed. Under the condition that there is no self-contradiction, the various embodiments can be freely combined to form new embodiments. That is, the embodiments belonging to the replacement schemes can be freely replaced but cannot be combined with each other; the embodiments that do not belong to the replacement schemes can be combined with each other, and these new embodiments also belong to the essential content of the present invention.

[0248] The above embodiments describe multiple specific implementations of the present invention, but those skilled in the art should understand that various changes or modifications can be made to these implementations without departing from the principles and essence of the present invention, but these changes and modifications are all within the scope of protection of the present invention.

Claims

1. A multi-person rope skipping counting method based on target recognition, characterized in that: include: S1. Multi-source data acquisition and synchronization: The camera collects color image signals, the microphone array collects voice command signals, and the gyroscope collects attitude signals. These signals are then timestamped and aligned to produce a multi-source signal with aligned timestamps. S2. Image Preprocessing and Target Recognition: Perform bilinear interpolation scaling and normalization on the color image signals from the multi-source signals to obtain a preprocessed image. Use the YOLOv5 model to perform target recognition on the preprocessed image and obtain the human target coordinate signal. S3. Posture Processing and Trajectory Fitting: Kalman filtering is performed on the posture signals from the multi-source signals. The terminal rotation angle increment is calculated by integration to generate a jitter compensation signal. Based on the human target coordinate signal and the jitter compensation signal, a surface fitting is performed using a sine function superimposed on linear terms to generate a smooth jump trajectory signal. The first-order derivative of the smooth jump trajectory signal is calculated to obtain a trajectory velocity derivative signal. The maximum value of the current cycle is extracted from the trajectory velocity derivative signal to obtain the trajectory velocity peak feature. S4. Speech Semantic Parsing and Strategy Generation: This involves performing noise reduction and speech recognition on multi-source speech command signals to generate semantic text. A lightweight large language model is used to parse scene semantics and motion patterns. This is combined with the amplitude of the jitter compensation signal to generate a dynamic parameter adjustment signal containing a threshold adjustment factor and speed weight. S5. LSTM Timing Analysis and Probability Calculation: The smooth jump trajectory signal and trajectory velocity derivative signal are input into the LSTM network. The forget gate, input gate, and output gate mechanism are used to analyze the timing characteristics and output the jump probability signal. S6. Multi-source information fusion counting decision: The trajectory velocity peak feature is weighted according to the velocity weight in the dynamic parameter adjustment signal. The weighted trajectory velocity peak feature is multiplied by the jump probability signal. The result is then compared with the dynamic threshold calculated from the threshold adjustment factor in the dynamic parameter adjustment signal to generate a counting decision signal. When the decision condition is met, the corresponding area counter is incremented by 1. S7. Result visualization: The counting decision signal is superimposed on the original image to generate a visual counting result.

2. The method for counting multiple people skipping ropes based on target recognition according to claim 1 is characterized in that: The color image captured by the camera in S1 is an RGB signal with a resolution of 1920×1080. When it is scaled to 608×608 pixels through bilinear interpolation, a weighted average formula is used to calculate the new pixel value. After normalization, the pixel value range is [-1, 1].

3. The method for counting multiple people skipping ropes based on target recognition according to claim 1, characterized in that: The microphone array in the S1 is configured with an 8cm spacing, achieves ±3° sound source localization through a phase difference algorithm, and combines with a dynamic noise suppression filter to attenuate 90dB of ambient noise.

4. The method for counting multiple rope skipping based on target recognition according to claim 1, characterized in that: The processing of the attitude signal in S3 includes: fusing the three-axis angular velocity data of the previous 100ms through a Kalman filter, using a trapezoidal integration method to calculate the terminal rotation angle increment, and generating a jitter compensation signal for correcting the linear drift term of the surface fitting.

5. The method for counting multiple people skipping ropes based on target recognition according to claim 1, characterized in that: The training data of the YOLOv5 model in S2 contains 20,000 images of multiple people skipping rope, covering scenes with 1 to 50 people, users aged 7 to 50, and sunny / cloudy / indoor lighting conditions. Mosaic data enhancement and cosine annealing learning rate strategy are used to optimize the model until the average precision calculated is greater than or equal to 0.96 when the intersection-over-union ratio threshold is 0.

5.

6. The method for counting multiple rope skipping based on target recognition according to claim 1, characterized in that: The mathematical model used for surface fitting in S3 is: , add regularization terms when solving parameters by least squares method , the RMS fitting error is less than 4 pixels, where For the Trajectory fitting function of the region, is the jump amplitude, which represents the user's jump height; is the angular frequency, , is the hopping frequency; is the phase shift, which represents the difference in jump starting posture; is the linear drift coefficient, compensating for the user's forward and backward movement; is the vertical reference, corresponding to the reference coordinate when the user is standing; is Gaussian noise, is the time variable, is the regularization coefficient.

7. The method for counting multiple rope skipping based on target recognition according to claim 6 is characterized in that: The lightweight large language model in S4 is built based on the LLaMA-7B architecture, and the input layer receives the amplitude of the jump trajectory , angular frequency , linear drift coefficient and semantic text embedding vectors, the threshold adjustment factor generated by the output layer ranges from [-0.3, 0.3], and the speed weight range is [0, 1].

8. The method for counting multiple rope skipping based on target recognition according to claim 1, characterized in that: The generation rule of the counting decision signal in S6 is: when the jump probability × (1 + speed weight × speed peak / 30) > dynamic threshold, it is determined to be a valid jump, where: The velocity peak is the maximum value of the trajectory velocity derivative signal in the current cycle; The dynamic threshold is calculated by the basic threshold × (1 + threshold adjustment factor), and the basic threshold is preset to 0.

6.

9. The method for counting multiple rope skipping based on target recognition according to claim 1, characterized in that: The specific implementation of surface fitting in S3 includes: Input 30 consecutive frames of human target coordinate signals into the Kalman filter and output a smooth coordinate sequence; perform fast Fourier transform on the smooth coordinate sequence and extract the main frequency component as the initial frequency parameter estimated value of; The parameters of the sinusoidal model are optimized by the nonlinear least squares method, and the iterative convergence condition is that the parameter change rate between two adjacent iterations is less than 0.01%.

10. The method for counting multiple rope skipping based on target recognition according to claim 1, characterized in that: The generation rule of the counting decision signal in S6 further includes: When the jump probability of three consecutive cycles is greater than 0.9, the fast confirmation mechanism is triggered and it is directly determined to be a valid jump; When the peak speed / reference speed is detected to be greater than 1.5, the double integration mode is activated and the corresponding area counter is incremented by 2; When the amplitude of the jitter compensation signal is greater than 0.8 rad / s, the anti-interference enhancement mode is activated and the dynamic threshold is increased by 50%.

Images