A data refinement and boxing action recognition method and system for a boxing scene

Abstract

The application discloses a boxing scene-oriented data refinement and boxing action recognition method and system, and belongs to the technical field of data mining; the method comprises the following steps: collecting time series data of multiple bone points of a boxer; detecting abnormality and loss of the time series data; processing the detected abnormal and lost time series data; constructing a boxing action recognition model based on an LSTM and a TCN network; inputting the processed time series data into the boxing action recognition model to realize a boxing action recognition task; the fuzzy C-means clustering algorithm and the projection approximation point algorithm are combined and used to reconstruct approximate real human body skeleton movement data; through comprehensive analysis on movement information of the whole human body skeleton including multiple bone points, accurate recognition and classification of boxing actions are realized, and the accuracy of action recognition is improved.

Description

Technical Field

[0001] This invention belongs to the field of data mining technology, specifically relating to a method and system for data refinement and boxing action recognition in boxing scenarios. Background Technology

[0002] As a high-intensity, fast-paced combat sport, boxing demands exceptional skill, strength, and strategy from its athletes. In boxing training, coaching guidance is crucial; accurate movement identification helps coaches promptly identify and correct technical errors, improving training effectiveness. However, boxing movements are complex and varied, including numerous offensive punches, defensive maneuvers, footwork, and feints. Traditional methods of manual observation and evaluation are inefficient, subjective, and unable to provide real-time feedback.

[0003] Traditional IMUs or other sensing methods involve attaching sensors to the boxer's wrist; some marking methods require attaching markers to the athlete's body. However, in a boxing match setting, these devices or methods not only affect the athlete's performance but can also cause the markers to detach or the data to become distorted due to collisions.

[0004] Traditional boxing movement recognition methods often focus only on the movements of the hands or a few key joints, neglecting the important role of other joints in boxing, such as the legs and torso. This method has certain limitations in recognizing complex boxing movements, such as footwork and body rotation.

[0005] In summary, existing algorithms have the following shortcomings:

[0006] 1. Traditional IMU or other sensing methods have limitations in boxing match scenarios and cannot accurately identify movements;

[0007] 2. Traditional manual observation and evaluation methods are easily influenced by personal biases, leading to subjective evaluation results.

[0008] 3. Traditional boxing motion recognition methods only focus on the movement of the hands or a few key joints, ignoring the important role of other joints in boxing, such as the legs and torso.

[0009] 4. Traditional IMU or other sensing methods may affect the athlete's performance in a boxing match and may cause markers to fall off or data to be distorted.

[0010] Therefore, how to solve the problems of data distortion and missing data in action recognition and improve the accuracy of action recognition is a key issue. Summary of the Invention

[0011] The purpose of this invention is to provide a data refinement and boxing action recognition system and method for boxing scenarios, in order to solve the problems mentioned in the background art.

[0012] The objective of this invention is achieved as follows: a method for data refinement and boxing motion recognition in boxing scenarios, characterized by the following steps:

[0013] Step S1: Collect time series data of multiple skeletal points of a boxer;

[0014] Step S2: Perform anomaly and missing data detection on the time series data;

[0015] Step S3: Process the detected abnormal and missing time series data;

[0016] Step S4: Construct and train a boxing motion recognition model based on LSTM and TCN networks;

[0017] Step S5: Input the processed time series data into the boxing motion recognition model to achieve the boxing motion recognition task.

[0018] Preferably, in step S1, the time series data of multiple skeletal points of the boxer is collected, and the specific operation is as follows:

[0019] Step S1-1: Four cameras are set up around the boxing ring to capture real-time video streams;

[0020] Step S1-2: Use the OpenPose deep learning algorithm to obtain the corresponding two-dimensional coordinate sequence of multi-bone points in the video stream;

[0021] Step S1-3: Using the pinhole imaging principle, the two-dimensional coordinates on the imaging plane are projected onto the three-dimensional coordinates in space to obtain the three-dimensional coordinates of the multi-skeletal points;

[0022] Step S1-4: Convert the three-dimensional coordinates of the multi-skeletal points into time series data.

[0023] Preferably, in step S2, anomaly and missing data detection is performed on the time series data, specifically as follows:

[0024] Step S2-1: Represent the multi-skeletal point time series data in matrix form, as follows:

[0025] F = [f1, f2, ..., f T ];

[0026] Among them, f i =[x i,1 ,y i,1 ,z i,1 ,...,x i,K ,yi,K ,z i,K ] T ,i=1,...,T,t is the total number of frames in the time series, andK is the number of skeleton points;

[0027] Step S2-2: Define anomalies and missing data in time series data;

[0028] Data missing refers to the fact that in a multivariate time series, some timestamps of data are not recorded or are missing, which is manifested as some timestamps in matrix F being null values;

[0029] Data anomalies refer to the significant deviation of certain variables from the normal pattern at certain timestamps in a multivariate time series, which manifests as oscillations in certain data in matrix F or outliers in data over a certain period of time.

[0030] Step S2-3: Label outliers and missing data in the time series data;

[0031] If there are missing or outlier data, the labels for multivariate time series samples are [1,1].

[0032] If there are no missing or outliers in the data, the labels for the multivariate time series samples are [0,0].

[0033] Preferably, in step S3, the detected abnormal and missing time series data are processed as follows:

[0034] Step S3-1: For long-term sequences with different action combinations, fuzzy C-means clustering is used to divide the long sequences, so that the same or similar motion sequences can be grouped into the same segment:

[0035] For a matrix F = [f1, f2, ..., f T Based on the membership degree w of each data point to each class, slice f is created. i (i = 1, ..., w) are divided into c fuzzy groups, and the membership matrix S takes values ​​in the range [0, 1]. After normalization, the sum of the membership degrees of a dataset equals 1, that is:

[0036]

[0037] Among them, s ij This represents the membership degree of slice j to cluster i. The closer the value is to 1, the more the slice belongs to cluster i.

[0038] The value function of the fuzzy C-means is:

[0039]

[0040] Among them, 0 <sij <1,c i Let d be the cluster center of fuzzy group i. ij =||c i -f j || represents the Euclidean distance between the i-th cluster center and the j-th slice, and m is the fuzziness exponent, where m > 1;

[0041] A new objective function is constructed to ensure that the value function can be optimized along with the constraints. The new objective function is shown in the following equation:

[0042]

[0043] The necessary condition for the value function to be minimized by differentiating it with respect to the input is:

[0044]

[0045] Where, x j For time-series data; initialize the membership matrix S with values ​​ranging from [0,1], and satisfy... Conditions, calculated by cluster center c i The value function is calculated, and S is iteratively updated until the cluster center c that minimizes the value function is obtained. i And the membership matrix S;

[0046] Step S3-2: Reconstruct all elements in the recovery matrix based on the low-rank property of the data matrix, transforming the filling problem into a convex optimization problem;

[0047] The convex optimization problem is:

[0048]

[0049] Where X is the recovered matrix, M is the missing data matrix, Ω is the index set of known elements, and P... Ω It is a linear projection operator;

[0050]

[0051] Step S3-3: Solve the convex optimization problem using the projection approximation point algorithm;

[0052] Step S3-4: Obtain the complete multi-skeletal point temporal motion data after filling using the growing Lagrange multiplier algorithm.

[0053] Preferably, in step S3-3, the convex optimization problem is solved using the projection approximation point algorithm, specifically as follows:

[0054] For the missing data matrix of each sub-segment Where t wLet w be the number of frames in the w-th segment of the sequence after clustering, and d be the data dimension, where d = 3K.

[0055] Step S3-3-1: Set initialization parameters: X0 = 0, V0 = 0, n = 1.

[0056] Step S3-3-2: Calculate the projection matrix Y of the missing data matrix M. n The iterative formula is

[0057] Step S3-3-3: Project the matrix Y n Perform singular value decomposition and solve for the approximate point matrix X. n ,Right now:

[0058] [U n ,S n V n ] = SVD(Y n );

[0059]

[0060] in, The contraction operator has the following operation rules:

[0061]

[0062] pass For matrix S n Perform a contraction operation;

[0063] Step S3-3-4: During the solution process of the projection approximation point algorithm, the following convergence conditions are set:

[0064]

[0065] Convergence condition is and ε is the convergence threshold; the convergence of the low-rank matrix X is measured by the first convergence condition. n The reconstruction error of the original matrix M is used to determine the low-rank matrix X in the current iteration. n Whether it is close enough to the observation matrix M; the second convergence condition measures the difference between the results of two iterations, indicating the degree of convergence of the matrix update;

[0066] In steps S3-4, the complete multi-skeletal point temporal motion data after filling is obtained according to the growing Lagrange multiplier algorithm, specifically as follows:

[0067] According to the growing Lagrange multiplier algorithm, V n The iterative update formula is:

[0068] When the convergence condition is met, or when n exceeds the threshold of the number of operations n. max The solution X is obtained n The optimal solution yields the complete multi-bone point temporal motion data X after filling.

[0069] Preferably, the action recognition model includes a Long Short-Term Memory (LSTM) module, a multi-head attention mechanism module, and a temporal convolutional module (TCN). The LSTM module is used to maintain the hidden vector h and the memory vector m, which respectively control the state update and input of each timestamp, capturing the temporal dependencies of multivariate time series.

[0070] The Long Short-Term Memory (LSTM) module includes a forget gate, an input gate, and an output gate. The forget gate determines how much of the cell state from the previous time step is retained at the current time step c. t The input gate determines the network input x at the current moment. t How much is saved to cell state c? t Output gate control unit status c t How much output is given to the current output value h of the LSTM? t ;

[0071] The input gate includes three inputs, namely the network's input value x at the current time. t The output value h of the LSTM at the previous time step t-1 and the cell state c from the previous time step t-1 ;

[0072] The output gate includes two outputs, the current LSTM output value h. t and the cell state c from the previous time step t-1 ;

[0073] The multi-head attention mechanism module includes a linear transformation module, a self-attention calculation module, and a concatenation module. The linear transformation module performs a linear transformation to obtain the query, key, and value of multiple heads.

[0074] Q i =QW i q K i =KW i k V i =VW i v ;

[0075] Among them, Q i Let K be the query matrix for the i-th head. i Let V be the key matrix of the i-th head. i This is the value matrix of the i-th head; l is the sequence length, n is the feature dimension of the TCN output; Q is the query, K is the key, and V is the value;

[0076] The self-attention calculation module performs self-attention calculations for each head:

[0077]

[0078] Where, n h The number of heads;

[0079] The outputs of all the heads are concatenated and then restored back to the input sequence dimension through a linear transformation:

[0080] MultiHead(Q,K,V)=Concat(head1,...,head h W O ;

[0081] in, It is a trainable linear transformation weight matrix;

[0082] The self-attention computation module performs independent self-attention operations on the input query q, key k, and value v through multiple heads, concatenates the results of all heads, and obtains the final output through a linear transformation. The multiple heads focus on different parts of the input sequence to varying degrees.

[0083] The temporal convolution module is composed of multiple layers of causal convolution kernels and extended convolutions stacked together, and the output of each layer is passed through residual connections;

[0084] Temporal Convolutional Networks (TCNs) use causal convolution kernels to model temporal data. They model long-term temporal data through residual connections and extended convolutions, capturing local temporal dependencies in time series.

[0085] For the input signal x t And a causal convolution kernel W, the output of the causal convolution kernel is:

[0086]

[0087] Among them, w i x represents the weights of the convolution kernel; t-i The timing data is input to the TCN;

[0088] Expanding the convolution increases the receptive field of the temporal convolutional network for temporal data, capturing dependencies over longer periods. The output of the expanded convolution is:

[0089]

[0090] Where r is the expansion factor, and the receptive field gradually expands as the number of layers increases.

[0091] Preferably, the action recognition model is trained by constructing a loss function, specifically as follows:

[0092] Step S4-1: Construct a smoothing loss function to ensure the smoothness of the motion sequence;

[0093] X is the matrix obtained after data refinement; repeating the boundary elements of X yields X′:X1′. ,i =X2′ ,i =X 1,i , X′ 3K+1,i =X′ 3K,i =X 3K,i , Where 1≤i≤T;

[0094] Define O as a symmetric tridiagonal matrix that satisfies the following conditions:

[0095]

[0096]

[0097] Where 2≤j≤t-1, Since the interval between frames is equal, when 1≤j≤T, h j =1, then

[0098]

[0099] Step S4-2: Define the smoothing loss function, which is:

[0100]

[0101] Where T is the number of frames in the sequence;

[0102] Step S4-3: Based on the prior knowledge of the kinematic model, introduce the bone length loss function, which is:

[0103] set {l b |1≤b≤J-1} is the bone length sequence, where b is the bone index, and the bone length loss is:

[0104]

[0105] in, and Let be the three-dimensional position of the joints at both ends of the bone in the i-th frame;

[0106] Step S4-4: Introduce cross-entropy loss to evaluate and optimize the action recognition model;

[0107] The cross-entropy loss function is used to calculate the difference between the probability distribution of each action category output by the classifier and the probability distribution corresponding to the true action label. The cross-entropy loss is expressed as:

[0108] Loss c =-∑ i q i log(p i );

[0109] Where, q i p represents the probability of the true label being in the i-th class. i This represents the probability that the classifier predicts for class i.

[0110] The total loss function of the action recognition model is:

[0111] Loss = λ1Loss s +λ2Loss b +λ3Loss c ;

[0112] Wherein, λ1, λ2, and λ3 are the weight coefficients of smoothing loss, bone length loss, and cross-entropy loss, respectively; the weight coefficients are used as hyperparameters during the training process and are fine-tuned according to the training conditions and the action recognition performance of the test set.

[0113] Preferably, in step S5, the processed time-series data is input into the boxing motion recognition model to achieve the boxing motion recognition task, specifically as follows:

[0114] Step S5-1: Split the processed time series data into training and test sets in an 8:2 ratio, normalize the time series data, and input the training set into the action recognition model;

[0115] Step S5-2: When the time series length T is greater than the number of variables 3K, the data is input to the Long Short-Term Memory (LSTM) module. The LSTM module provides the data of the variables at all T timestamps.

[0116] The dataset has dimensions of R. N×T×3K Where N is the number of samples in the dataset, T is the time series length of each sample, and K is the number of skeletal points;

[0117] Step S5-3: Local features of time series data are obtained through one-dimensional convolutional blocks of the temporal convolutional network TCN. Then, the mean and standard deviation of each batch of data are adjusted by batch normalization to make the distribution of time series data more stable.

[0118] Step S5-4: By introducing nonlinearity through the ReLU activation function, the action recognition model learns complex patterns and relationships in the data; the output sequence local features are processed through a multi-head attention mechanism to obtain long-range dependencies between features and capture the global context.

[0119] Step S5-5: The features output by the dual branches are aggregated by concat and the input vector is transformed into a probability distribution for each class by softmax, thus achieving the effect of sequence classification;

[0120] Steps S5-6: Use the Adam optimizer and backpropagate the cross-entropy loss function throughout the network.

[0121] A data refinement and boxing motion recognition system for boxing scenarios, characterized by:

[0122] The motion recognition system includes a labelless motion capture module, a data missing and anomaly detection module, a multi-skeletal point missing data filling module, a multi-skeletal point data repair module, and a boxing motion recognition module. The labelless motion capture module is used to collect multi-skeletal point time-series data of boxers in actual boxing training and competition scenarios, obtaining multi-skeletal point time-series motion data of boxers. The data missing and anomaly detection module is used to detect missing or anomalies in the output time-series data, and to label the data type by detecting whether there are missing or anomalies. The multi-skeletal point missing data filling module processes incomplete skeletal time-series data, and reconstructs data that approximates real human skeletal motion data through fuzzy C-means clustering algorithm and projection approximation point algorithm.

[0123] The multi-skeletal point data repair module realizes the anomaly removal and smoothing of multi-skeletal point temporal motion data, and the boxing action recognition module uses the boxing action recognition model to realize the boxing action recognition task.

[0124] Compared with the prior art, the present invention has the following improvements and advantages:

[0125] 1. By combining the fuzzy C-means clustering algorithm and the projection approximation point algorithm, we reconstruct motion data that approximates the real human skeleton. By comprehensively analyzing the motion information of the entire human skeleton, including multiple skeletal points, we can achieve accurate identification and classification of boxing movements and improve the accuracy of motion recognition.

[0126] 2. By employing different loss functions to smooth motion sequences and fine-tuning them on different datasets, the accuracy of action recognition is further improved. Attached Figure Description

[0127] Figure 1 This is a flowchart of the method of the present invention.

[0128] Figure 2This is a structural diagram of the system of the present invention.

[0129] Figure 3 This is a structural diagram of the action recognition model. Detailed Implementation

[0130] The invention will be further summarized below with reference to the accompanying drawings.

[0131] like Figure 1 As shown, a method for data refinement and boxing motion recognition in boxing scenarios includes the following steps:

[0132] Step S1: Collect time series data of multiple skeletal points of a boxer;

[0133] Step S1-1: Four cameras are set up around the boxing ring to capture real-time video streams;

[0134] Four cameras are connected to the markerless motion capture module via a synchronization interface. The motion capture module supports one-click synchronized shooting. To ensure the stability of the time series output by the markerless motion capture module, one master camera is selected from the four cameras and connected to the motion capture module. The master camera is connected to the other slave cameras via a dedicated synchronization line to achieve signal synchronization. The user sends a trigger signal to the master camera by pressing a button. After receiving the signal, the master camera immediately transmits the signal to all slave cameras through the synchronization line, so that the four cameras can start and stop shooting at the same time, ensuring that the start and end times of shooting are completely synchronized.

[0135] Step S1-2: Use the OpenPose deep learning algorithm to obtain the corresponding two-dimensional coordinate sequence of multi-bone points in the video stream;

[0136] OpenPose can identify and locate athletes' body postures and movement trajectories in videos, extracting key joints such as the head, shoulders, elbows, wrists, hips, knees, and ankles. For the i-th camera (i = 1, 2, 3, 4), the algorithm can obtain the two-dimensional coordinates of multiple skeletal points on the imaging plane in the video captured by the i-th camera. OpenPose supports real-time video stream input and accurately locates key joints in each frame of the video.

[0137] Step S1-3: Using the pinhole imaging principle, the two-dimensional coordinates on the imaging plane are projected onto the three-dimensional coordinates in space to obtain the three-dimensional coordinates of the multi-skeletal points;

[0138] For a certain key point P in space, (u i ,v i Let P be the two-dimensional coordinates of the i-th camera's imaging plane. The image on the camera can be represented using the pinhole camera model, which is expressed as:

[0139]

[0140]

[0141] Among them, f x and f y These represent the camera's horizontal and vertical focal lengths, respectively; u0 and v0 are the pixel coordinates of the intersection of the camera's optical axis and the imaging plane, respectively; f x f y u0 and v0 are the camera's internal parameters, M i Let x be the extrinsic parameter matrix from the world coordinate system to the camera i coordinate system. w y w , z w Let z be the three-dimensional coordinates of the joint point P in space. c This is the scaling factor, which defaults to 1.

[0142] For the i-th camera, the three-dimensional coordinates P are derived from the above formula. i (X w ,Y w Z w The mapping relationship between the point and its two-dimensional plane coordinates as imaged by a camera:

[0143]

[0144] in,

[0145] Step S1-4: Convert the 3D coordinates of the multi-skeletal points into time series data;

[0146] The above mapping relationship applies to four cameras, and the rays emitted by the four cameras can intersect at a single point, namely P(x). w ,y w ,z w However, due to the unavoidable noise in the data, ray l i The deviation causes multiple rays to not intersect at a single point, so it is necessary to estimate the spatial position of the key point P. Considering the distance between the rays, a spherical region is set, with the radius of the sphere being 10 times the shortest distance between the rays. The moving point P' moves within the spherical region. Combining the OpenPose confidence parameter, the spatial point with the smallest sum of distances to multiple rays is taken as the intersection of the multiple rays, and this spatial point is taken as the spatial position of the key point P.

[0147] The OpenPose deep learning algorithm outputs a two-dimensional coordinate sequence. Through the above process, a three-dimensional multivariate time series covering the motion information of multiple joints of the human body can be obtained, i.e., time series data. The variable of the multivariate time series is a certain dimension of a certain joint.

[0148] Step S2: Perform anomaly and missing data detection on the time series data, specifically as follows:

[0149] Step S2-1: Represent the multi-skeletal point time series data in matrix form, as follows:

[0150] F = [f1, f2, ..., f T ];

[0151] Among them, f i =[x i,1 ,y i,1 ,z i,1 ,...,x i,K ,y i,K ,z i,K ] T ,i=1,...,T,t is the total number of frames in the time series, andK is the number of skeleton points;

[0152] Step S2-2: Define anomalies and missing data in time series data;

[0153] Data missing refers to the fact that in a multivariate time series, some timestamps of data are not recorded or are missing, which is manifested as some timestamps in matrix F being null values;

[0154] Data anomalies refer to the significant deviation of certain variables from the normal pattern at certain timestamps in a multivariate time series, which manifests as oscillations in certain data in matrix F or outliers in data over a certain period of time.

[0155] Step S2-3: Label outliers and missing data in the time series data;

[0156] If there are missing or outlier data, the labels for multivariate time series samples are [1,1].

[0157] If there are no missing or outliers in the data, the labels for the multivariate time series samples are [0,0].

[0158] Step S3: Process the detected anomalies and missing time series data, specifically as follows:

[0159] Step S3-1: For long-term sequences with different action combinations, fuzzy C-means clustering is used to divide the long sequences, so that the same or similar motion sequences can be grouped into the same segment:

[0160] For a matrix F = [f1, f2, ..., f T Based on the membership degree w of each data point to each class, slice f is created. i(i = 1, ..., w) are divided into c fuzzy groups, and the membership matrix S takes values ​​in the range [0, 1]. After normalization, the sum of the membership degrees of a dataset equals 1, that is:

[0161]

[0162] Among them, s ij This represents the membership degree of slice j to cluster i. The closer the value is to 1, the more the slice belongs to cluster i.

[0163] The value function of the fuzzy C-means is:

[0164]

[0165] Among them, 0 ij <1,c i Let d be the cluster center of fuzzy group i. ij =||c i -f j || represents the Euclidean distance between the i-th cluster center and the j-th slice, and m is the fuzziness exponent, where m > 1;

[0166] A new objective function is constructed to ensure that the value function can be optimized along with the constraints. The new objective function is shown in the following equation:

[0167]

[0168] The necessary condition for the value function to be minimized by differentiating it with respect to the input is:

[0169]

[0170] Where, x j For time-series data; initialize the membership matrix S with values ​​ranging from [0,1], and satisfy... Conditions, calculated by cluster center c i The value function is calculated, and S is iteratively updated until the cluster center c that minimizes the value function is obtained. i And the membership matrix S;

[0171] Step S3-2: Reconstruct all elements in the recovery matrix based on the low-rank property of the data matrix, transforming the filling problem into a convex optimization problem;

[0172] The convex optimization problem is:

[0173]

[0174] Where X is the recovered matrix, M is the missing data matrix, Ω is the index set of known elements, and P... Ω It is a linear projection operator; ​

[0175]

[0176] Step S3-3: Solve the convex optimization problem using the projection approximation point algorithm;

[0177] For the missing data matrix of each sub-segment Where t w Let w be the number of frames in the w-th segment of the sequence after clustering, and d be the data dimension, where d = 3K.

[0178] Step S3-3-1: Set initialization parameters: X0 = 0, V0 = 0, n = 1.

[0179] Step S3-3-2: Calculate the projection matrix Y of the missing data matrix M. n The iterative formula is

[0180] Step S3-3-3: Project the matrix Y n Perform singular value decomposition and solve for the approximate point matrix X. n ,Right now:

[0181] [U n ,S n V n ] = SVD(Y n );

[0182]

[0183] in, The contraction operator has the following operation rules:

[0184]

[0185] pass For matrix S n Perform a contraction operation;

[0186] Step S3-3-4: During the solution process of the projection approximation point algorithm, the following convergence conditions are set:

[0187]

[0188] Convergence condition is and ε is the convergence threshold; the convergence of the low-rank matrix X is measured by the first convergence condition. n The reconstruction error of the original matrix M is used to determine the low-rank matrix X in the current iteration. n Whether it is close enough to the observation matrix M; the second convergence condition measures the difference between the results of two iterations, indicating the degree of convergence of the matrix update;

[0189] Step S3-4: Obtain the complete multi-skeletal point temporal motion data after filling using the growing Lagrange multiplier algorithm.

[0190] According to the growing Lagrange multiplier algorithm, V n The iterative update formula is:

[0191] When the convergence condition is met, or when n exceeds the threshold of the number of operations n. max The solution X is obtained n The optimal solution yields the complete multi-bone point temporal motion data X after filling.

[0192] Step S4: Construct and train a boxing motion recognition model based on LSTM and TCN networks;

[0193] The action recognition model includes a Long Short-Term Memory (LSTM) module, a multi-head attention mechanism module, and a temporal convolution module (TCN). The LSTM module is used to maintain the hidden vector h and the memory vector m, which control the state update and input of each timestamp, respectively, and capture the temporal dependencies of multivariate time series.

[0194] The Long Short-Term Memory (LSTM) module includes a forget gate, an input gate, and an output gate. The forget gate determines how much of the cell state from the previous time step is retained in the current time step c. t The input gate determines the network input x at the current moment. t How much is saved to cell state c? t Output gate control unit status c t How much output is given to the current output value h of the LSTM? t ;

[0195] The input gate has three inputs: the network's current input value x. t The output value h of the LSTM at the previous time step t-1 and the cell state c from the previous time step t-1 ;

[0196] The output gate has two outputs: the current LSTM output value h. t and the cell state c from the previous time step t-1 ;

[0197] The multi-head attention mechanism module includes a linear transformation module, a self-attention calculation module, and a concatenation module. The linear transformation module performs a linear transformation to obtain the query, key, and value of multiple heads.

[0198] Q i =QW i q K i =KW ik V i =VW i v ;

[0199] Among them, Q i Let K be the query matrix for the i-th head. i Let V be the key matrix of the i-th head. i This is the value matrix of the i-th head; l is the sequence length, n is the feature dimension of the TCN output; Q is the query, K is the key, and V is the value;

[0200] The self-attention calculation module performs self-attention calculations for each head:

[0201]

[0202] Where, n h The number of heads;

[0203] The outputs of all the heads are concatenated and then restored back to the input sequence dimension through a linear transformation:

[0204] MultiHead(Q,K,V)=Concat(head1,...,head h W O ;

[0205] in, It is a trainable linear transformation weight matrix;

[0206] The self-attention computation module performs independent self-attention operations on the input query q, key k, and value v through multiple heads, concatenates the results of all heads, and obtains the final output through a linear transformation. The multiple heads focus on different parts of the input sequence to varying degrees.

[0207] The temporal convolution module consists of multiple layers of causal convolution kernels and extended convolutions stacked together, with the output of each layer being passed through residual connections;

[0208] Temporal Convolutional Networks (TCNs) use causal convolution kernels to model temporal data. They model long-term temporal data through residual connections and extended convolutions, capturing local temporal dependencies in time series.

[0209] For the input signal x t And a causal convolution kernel W, the output of the causal convolution kernel is:

[0210]

[0211] Among them, w i x represents the weights of the convolution kernel; t-i The timing data is input to the TCN;

[0212] Expanding the convolution increases the receptive field of the temporal convolutional network for temporal data, capturing dependencies over longer periods. The output of the expanded convolution is:

[0213]

[0214] Where r is the expansion factor, and the receptive field gradually expands as the number of layers increases.

[0215] The action recognition model is trained by constructing a loss function, specifically:

[0216] Step S4-1: Construct a smoothing loss function to ensure the smoothness of the motion sequence;

[0217] X is the matrix obtained after data refinement; repeating the boundary elements of X yields X′:X1′. ,i =X2′ ,i =X 1,i , X′ 3K+1,i =X′ 3K,i =X 3K,i , Where 1≤i≤T;

[0218] Define O as a symmetric tridiagonal matrix that satisfies the following conditions:

[0219]

[0220] Where 2≤j≤t-1, Since the interval between frames is equal, when 1≤j≤T, h j =1, then

[0221]

[0222] Step S4-2: Define the smoothing loss function, which is:

[0223]

[0224] Where T is the number of frames in the sequence;

[0225] Step S4-3: Based on the prior knowledge of the kinematic model, introduce the bone length loss function, which is:

[0226] set {l b |1≤b≤J-1} is the bone length sequence, where b is the bone index, and the bone length loss is:

[0227]

[0228] in, and Let be the three-dimensional position of the joints at both ends of the bone in the i-th frame;

[0229] Step S4-4: Introduce cross-entropy loss to evaluate and optimize the action recognition model;

[0230] The cross-entropy loss function is used to calculate the difference between the probability distribution of each action category output by the classifier and the probability distribution corresponding to the true action label. The cross-entropy loss is expressed as:

[0231] Loss c =-∑ i q i log(p i );

[0232] Where, q i p represents the probability of the true label being in the i-th class. i This represents the probability that the classifier predicts for class i.

[0233] The total loss function of the action recognition model is:

[0234] Loss = λ1Loss s +λ2Loss b +λ3Loss c ;

[0235] Wherein, λ1, λ2, and λ3 are the weight coefficients of smoothing loss, bone length loss, and cross-entropy loss, respectively; the weight coefficients are used as hyperparameters during the training process and are fine-tuned according to the training conditions and the action recognition performance of the test set.

[0236] Step S5: Input the processed time series data into the boxing motion recognition model to achieve the boxing motion recognition task, specifically:

[0237] Step S5-1: Split the processed time series data into training and test sets in an 8:2 ratio, normalize the time series data, and input the training set into the action recognition model;

[0238] Step S5-2: When the time series length T is greater than the number of variables 3K, the data is input to the Long Short-Term Memory (LSTM) module. The LSTM module provides the data of the variables at all T timestamps.

[0239] The dataset has dimensions of R. N×T×3K Where N is the number of samples in the dataset, T is the time series length of each sample, and K is the number of skeletal points;

[0240] Step S5-3: Local features of time series data are obtained through one-dimensional convolutional blocks of the temporal convolutional network TCN. Then, the mean and standard deviation of each batch of data are adjusted by batch normalization to make the distribution of time series data more stable.

[0241] Step S5-4: By introducing nonlinearity through the ReLU activation function, the action recognition model learns complex patterns and relationships in the data; the output sequence local features are processed through a multi-head attention mechanism to obtain long-range dependencies between features and capture the global context.

[0242] Step S5-5: The features output by the dual branches are aggregated by concat and the input vector is transformed into a probability distribution for each class by softmax, thus achieving the effect of sequence classification;

[0243] Steps S5-6: Use the Adam optimizer and backpropagate the cross-entropy loss function throughout the network.

[0244] A motion recognition system for boxing scenarios, based on data refinement and a method for generating boxing motion recognition, is disclosed. The system includes a labelless motion capture module, a data missing and anomaly detection module, a multi-skeletal point missing data filling module, a multi-skeletal point data repair module, and a boxing motion recognition module. The labelless motion capture module is used to collect multi-skeletal point time-series data of boxers in actual boxing training and competition scenarios, obtaining multi-skeletal point time-series motion data of boxers. The data missing and anomaly detection module is used to detect missing or anomaly in the output time-series data, labeling the data type by detecting the presence of missing or anomalies. The multi-skeletal point missing data filling module processes incomplete skeletal time-series data, reconstructing near-realistic human skeletal motion data using fuzzy C-means clustering and projection approximation point algorithms.

[0245] The multi-skeletal point data repair module realizes the anomaly removal and smoothing of multi-skeletal point temporal motion data, while the boxing action recognition module uses a boxing action recognition model to realize the boxing action recognition task.

[0246] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A data refinement and boxing action recognition method for a boxing scene, characterized in that: The method comprises the following steps: Step S1: collecting a plurality of bone point time series data of a boxer; Step S1-1: arranging four cameras around a boxing ring, and acquiring real-time video streams by using the cameras; Step S1-2: acquiring corresponding multi-bone point two-dimensional coordinate sequences in the video streams by using an OpenPose deep learning algorithm; Step S1-3: realizing projection of two-dimensional coordinates on an imaging plane to three-dimensional coordinates in space by a pinhole imaging principle, and obtaining multi-bone point three-dimensional coordinates; For the i-th camera, a mapping relationship between the three-dimensional coordinate point Pi (Xw, Yw, Zw) and the two-dimensional plane coordinates of the point imaged in the camera is obtained: ； wherein, ; (u i ,v i ) is the two-dimensional coordinate of P in the imaging plane of the i-th camera, z c is a scale factor, f x and f y are the horizontal and vertical focal lengths of the camera, respectively; u0, v0are the pixel coordinates of the intersection of the camera optical axis and the imaging plane, M i is the extrinsic matrix from the world coordinate system to the camera i coordinate system; Step S1-4: converting the multi-bone point three-dimensional coordinates into time series data; The mapping relationship is used on four cameras, and the rays emitted by the four cameras intersect at a point, that is ; Due to the existence of noise in the data, the rays deviate from each other, resulting in multiple rays not intersecting at a point, so the spatial position of the joint P needs to be estimated; considering the distance between the rays, a spherical region is set, the radius of the sphere is 10 times the shortest distance between the rays, and the moving point moves in the spherical region; combined with the OpenPose confidence parameter, the spatial point with the minimum distance to multiple rays in space is taken as the intersection of multiple rays, and this spatial point is taken as the spatial position of the joint P; Step S2: detecting abnormalities and missing in the time series data; Step S3: processing the detected abnormal and missing time series data; Step S4: constructing a boxing action recognition model based on an LSTM and a TCN network, and training the boxing action recognition model; Step S5: inputting the processed time series data into the boxing action recognition model to realize a boxing action recognition task; In the step S5, the processed time series data is input into the boxing action recognition model to realize the boxing action recognition task, specifically: Step S5-1: splitting the processed time series data into a training set and a test set according to a ratio of 8:2, normalizing the time series data, and inputting the training set into the action recognition model; Step S5-2: When the time series length T is greater than the variable number 3K, input to the long short-term memory module LSTM, and the data set dimension is Wherein, N is the number of samples in the data set, T is the time series length of each sample, and K is the number of bone points. Step S5-3: acquiring local features of the time series data by a one-dimensional convolution block of a time domain convolution network TCN, and then adjusting the mean and standard deviation of each batch data by batch normalization to make the time series data distribution more stable; Step S5-4: introducing nonlinearity by a ReLU activation function, learning complex patterns and relationships in the data by the action recognition model, acquiring long-range dependency relationships between features by a multi-head attention mechanism, and capturing global context; Step S5-5: aggregating the features output by the double branches by concat, and converting the input vector into a probability distribution of each class by softmax to achieve sequence classification; Step S5-6: using an Adam optimizer and a cross-entropy loss function to backpropagate the entire network.

2. The method of claim 1, wherein the method is a boxing scene oriented data refinement and boxing action recognition method. In the step S2, the time series data is detected for abnormalities and missing, specifically: Step S2-1: representing the multi-bone point time series data in a matrix form as follows: ； wherein, T is the total number of frames in the time series, and K is the number of skeletal points. Step S2-2: defining abnormalities and missing of the time series data; Data missing refers to that in the multivariate time series, data of some time stamps is not recorded or lost, which is shown as that some time stamps in the matrix F are null values; Data abnormality refers to that in the multivariate time series, data of some variables in some time stamps significantly deviates from the normal mode, which is shown as that some data in the matrix F oscillates or data in a time length deviates from the normal mode; Step S2-3: sample labeling of abnormalities and missing of the time series data; If the data has missing and abnormal conditions, the multivariate time series sample labeling label is [1, 1]. If the data is not missing and abnormal, the multi-variable time series sample label is marked as [0, 0].

3. The method of claim 1, wherein the method is a boxing scene oriented data refinement and boxing action recognition method. The step S3 is specifically as follows: Step S3-1: For long time series of different action combinations, a fuzzy C-means clustering method is used to divide the long sequence, so that the same or similar motion sequence can be classified into the same segment: For matrix According to the membership degree of each data point belonging to each class w slices Is divided into c fuzzy groups, the membership matrix S is in the range of [0, 1], after normalization processing, the sum of the membership degrees of a data set is equal to 1, that is: ； wherein, represents the membership of slice j to cluster i, the value closer to 1 indicates that this slice is more affiliated to cluster i; The value function of fuzzy C-means is: ； wherein , is the cluster center of the fuzzy group i, is the Euclidean distance between the i-th cluster center and the j-th slice, and m is the fuzziness index, ; A new objective function is constructed to ensure that the value function can be optimized with the constraint condition, and the new objective function is as follows: ； The necessary condition for the value function to reach the minimum is that the derivative of the input is: ； ； Wherein, is the time series data; the membership matrix S is initialized by the value range [0, 1], and satisfies The condition is that the clustering center is calculated And the value function is calculated, the S is iteratively updated, and finally the clustering center is obtained when the value function is minimum And the membership matrix S; Step S3-2: According to the low rank of the data matrix, all elements in the recovery matrix are recovered, and the filling problem is converted into a convex optimization problem; The convex optimization problem is: ； where X is the recovered matrix, M is the missing data matrix, is the index set of known elements, is the linear projection operator; ； Step S3-3: The convex optimization problem is solved by the projection approximation point algorithm; Step S3-4: According to the growing Lagrange multiplier algorithm, the complete multi-skeletal point time sequence motion data after filling is obtained.

4. The data refinement and boxing action recognition method for a boxing scene according to claim 3, characterized in that: The step S3-3 is specifically as follows: Missing data matrix for each sub-fragment wherein is the number of sequence frames in the w-th segment after clustering segmentation, d is the data dimension, d = 3K; Step S3-3-1: Set initialization parameters , , , ; Step S3-3-2: Calculate the projection matrix of the missing data matrix M , the iterative formula is ; Step S3-3-3: Singular value decomposition of the projected matrix Singular value decomposition and solving the approximation point matrix i.e.: ； ； wherein is a contraction operator with the following operation rule: ； By performing a contraction operation on the matrix ; Step S3-3-4: In the solving process of the projection approximation point algorithm, the following convergence condition is set: ； ； The convergence condition is and , is a convergence threshold; through the first convergence condition, the reconstruction error of the low-rank matrix and the original matrix is measured to determine whether the low-rank matrix of the current iteration is close enough to the observation matrix ; through the second convergence condition, the difference between the results of two iterations is measured to represent the convergence degree of matrix updating; Step S3-4 is specifically as follows: According to the growth Lagrange multiplier algorithm, The iterative update formula is ; When the convergence condition is reached, or n exceeds a threshold of the number of operations , the obtained solution is an optimal solution, and the filled complete multi-skeletal point timing motion data is obtained .

5. The data refinement and boxing action recognition method for boxing scene according to claim 1, characterized in that: The action recognition model includes a long short-term memory module LSTM, a multi-head attention mechanism module and a time domain convolution module TCN, the long short-term memory module LSTM is used to maintain a hidden vector h and a memory vector m, which respectively control the state update and input of each timestamp, and capture the time dependence of the multi-variable time series; The long short-term memory module LSTM includes a forgetting gate, an input gate, and an output gate, the forgetting gate determines how much of the cell state of the previous time is retained to the current time , the input gate determines how much of the input of the current time of the network is saved to the cell state , and the output gate controls how much of the cell state is output to the current output value of the LSTM , the output gate controls how much of the cell state is output to the current output value of the LSTM , the output gate controls how much of the cell state is output to the current output value of the LSTM ​ The input gate includes three inputs, respectively, an input value of a network at a current moment , an output value of an LSTM at a previous moment , and a cell state at the previous moment ; The output gate includes two outputs, an output value of the current time point LSTM and a cell state of the previous time point ; The multi-head attention mechanism module includes a linear transformation module, a self-attention calculation module and a splicing module, the linear transformation module performs linear transformation to obtain queries, keys and values of multiple heads: ， ， ； wherein, is a query matrix for the i-th head, is a key matrix for the i-th head, is a value matrix for the i-th head; , , , is a sequence length, is a feature dimension of the TCN output; is a query, is a key, is a value; The self-attention calculation module performs self-attention calculation on each head: ； wherein, is the number of heads; Splicing splices the outputs of all heads, and restores the input sequence dimension through linear transformation: ； wherein, is a trainable linear transformation weight matrix; The self-attention calculation module performs independent self-attention operation on the input query q, key k and value v through multiple heads, splices the results of all heads and performs linear transformation to obtain the final output, and different parts of the input sequence are paid attention to by multiple heads to different degrees; The time domain convolution module is stacked by multiple layers of causal convolution kernels and extended convolution, and the output of each layer is transmitted through a residual connection; The time domain convolution network TCN models the time series data by using the causal convolution kernel, models the long time series data by using the residual connection and extended convolution, and captures the local time dependence of the time series; For an input signal and a causal convolution kernel W, the output of the causal convolution kernel is: ； wherein, is a weight of a convolution kernel; is time series data input to the TCN; The extended convolution increases the receptive field of the time domain convolution network for the time series data, and captures the longer time dependence, and the output of the extended convolution is: ； wherein, is the expansion factor, and the receptive field gradually expands as the number of layers increases.

6. The data refinement and boxing action recognition method for a boxing scene according to claim 5, characterized in that: The training of the action recognition model is performed by constructing a loss function, and the loss function is specifically as follows: Step S4-1: Construct a smoothing loss function to ensure the smoothness of the motion sequence; is the matrix obtained after refining the data; repeat boundary elements of the matrix , , ; wherein ; Define O as a symmetric tri-diagonal matrix, which satisfies the following conditions: ； ； ； wherein , , ; since the intervals between frames are equal, when , , then ； Step S4-2: Define a smoothing loss function, and the smoothing loss function is: ； Wherein, T is the number of frames in the sequence; Step S4-3: According to the prior knowledge of the kinematic model, a bone length loss function is introduced, and the bone length loss function is: set is a sequence of bone lengths, where b is a bone index, and the bone length loss is: ； wherein, with is the three-dimensional position of the joint at the end of the bone in the i-th frame; Step S4-4: Cross-entropy loss is introduced to evaluate and optimize the action recognition model. The cross-entropy loss function is used to calculate the difference between the probability distribution of each action category output by the classifier and the probability distribution corresponding to the true action label, and the cross-entropy loss is represented as: ； wherein, denotes the probability of the true label being of class i, denotes the probability of the classifier predicting class i; The total loss function of the action recognition model is: ； wherein, , , are weight coefficients of the smooth loss, the bone length loss, and the cross-entropy loss, respectively; the weight coefficients are hyperparameters in the training process, which are fine-tuned along with the training and the action recognition performance of the test set.

7. A data refinement and boxing action recognition system for a boxing scene, for implementing the data refinement and boxing action recognition method of any one of claims 1-6, characterized in that: The action recognition system comprises an unlabeled motion capture module, a data loss and anomaly detection module, a multi-skeleton point missing data filling module, a multi-skeleton point data repair module, and a boxing action recognition module. The unlabeled motion capture module is used for collecting multi-skeleton point time series data of boxers in actual training and competition scenes of boxing, and obtaining multi-skeleton point time series motion data of boxers. The data loss and anomaly detection module is used for detecting the output time series data loss or anomaly, and labeling the data type label by detecting whether there is loss or anomaly. The multi-skeleton point missing data filling module is used for incomplete skeleton time series data processing, and reconstructs the approximate real human skeleton motion data by fuzzy C-means clustering algorithm and projection approximation point algorithm. The multi-skeleton point data repair module realizes the abnormal elimination and smoothing processing of multi-skeleton point time series motion data, and the boxing action recognition module realizes the boxing action recognition task by using the boxing action recognition model.

Images