Computer vision sports skill assessment method, computer device and storage medium

Abstract

The present application relates to a computer vision sports skill assessment method, computer equipment and storage medium. The method comprises: obtaining multiple initial video frames, obtaining multiple feature point displacement data; obtaining a background motion feature set; obtaining a global motion estimation matrix; obtaining a stable video data sequence; obtaining an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain a motion component signal; determining a screening factor for the motion component signal, and screening the motion component signal according to the screening factor to obtain an abnormal component signal; extracting features from the abnormal component signal to obtain a weighted feature vector; determining the weight corresponding to the abnormal component signal; determining the correction weight of the abnormal component signal, correcting the initial motion trajectory sequence to obtain a corrected motion trajectory sequence; comparing the corrected motion trajectory sequence with a preset standard action template to generate a comprehensive evaluation report. The present application can obtain accurate feedback on the action in a timely manner and improve the quality of training.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a computer vision sports skill evaluation method, computer equipment and storage medium. Background Art

[0002] In sports training, accurate assessment of students' sports skills and timely feedback are the key to improving training quality and efficiency. However, current skill assessment methods have many problems and limitations. Traditional assessment mainly relies on the coach's visual observation and experience judgment. This method is highly subjective and difficult to capture instantaneous subtle changes in movement. When observing multiple students, coaches often find it difficult to conduct accurate movement analysis and instant judgment at the same time, which not only increases their workload, but also may lead to inconsistencies and omissions in assessment. For students, they often find it difficult to accurately understand their movement problems through verbal feedback from their coaches, and they cannot obtain quantitative indicators to measure their skill level and room for improvement.

[0003] In addition, although some existing auxiliary tools such as high-speed cameras can record movement details, they often lack the ability to automatically analyze and evaluate. This requires coaches to spend a lot of time on manual analysis, which is inefficient. In a school environment, this problem is more prominent because teachers need to pay attention to the performance of multiple students at the same time and do not have enough time to analyze each student's movements in detail. At the same time, the training environment in schools is often not as stable as that in professional training venues, and there may be problems such as background interference and camera equipment shaking, which further increases the difficulty of accurate evaluation.

[0004] These problems seriously restrict the effectiveness of training and the speed of students' progress. Students cannot get accurate feedback on their movements in a timely manner, making it difficult to improve their technical movements in a targeted manner. Coaches find it difficult to effectively identify the individual differences and specific needs of each student and develop personalized training plans. In group training, this situation may cause some students' problems to be ignored, affecting the overall training quality. Summary of the invention

[0005] Based on this, it is necessary to provide a computer vision sports skills assessment method, computer equipment and storage medium that can accurately feedback Xu Sheng's movement status in response to the above technical problems.

[0006] A computer vision sports skill assessment method based on signal processing, comprising the following steps:

[0007] Collecting initial video data for student training, segmenting the initial video data to obtain multiple initial video frames, and calculating the displacement of feature points between adjacent initial video frames to obtain multiple feature point displacement data;

[0008] Extract the background motion features of the initial video frame according to the feature point displacement data to obtain a background motion feature set;

[0009] Perform temporal correlation analysis on feature point displacement data and background motion feature set to obtain a global motion estimation matrix;

[0010] Based on the global motion estimation matrix, motion compensation is performed on the initial video frame to obtain a stable video data sequence;

[0011] Performing human key point detection and tracking on the stable video data sequence to obtain an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain motion component signals;

[0012] Using the motion component signal, analyzing the position difference and time interval of adjacent key points of the human body, determining the screening factor of the motion component signal, and screening the motion component signal according to the screening factor to obtain the abnormal component signal;

[0013] Extract features of abnormal component signals to obtain weighted feature vectors; determine local weights corresponding to abnormal component signals based on the weighted feature vectors, screening factors, and initial motion trajectory sequences;

[0014] According to the screening factors and weights, the correction weights of the abnormal component signals are determined, and based on the correction weights, the initial motion trajectory sequence is corrected to obtain a corrected motion trajectory sequence;

[0015] The corrected motion trajectory sequence is compared with the preset standard action template, the deviation values ​​of the students' key points and limbs are calculated, and a comprehensive evaluation report is generated based on the deviation values ​​and preset scoring rules.

[0016] In one embodiment, the step of extracting background motion features of an initial video frame according to feature point displacement data to obtain a background motion feature set comprises the steps of:

[0017] Analyze the feature point displacement data based on the random sampling consistency algorithm to obtain the background motion feature model between each pair of adjacent initial video frames;

[0018] Detecting a set of background feature points from an initial video frame;

[0019] Based on the background motion feature model, predict the predicted position of the background feature point of the previous initial video frame in the next initial video frame between the corresponding adjacent initial video frames, and calculate the error between the predicted position and the actual position of the background feature point of the next initial video frame;

[0020] According to the error, a background motion feature vector corresponding to the initial video frame is defined, and a background motion feature set is constructed based on the background motion feature vector.

[0021] In one embodiment, the step of performing temporal correlation analysis on feature point displacement data and background motion feature set to obtain a global motion estimation matrix comprises the steps of:

[0022] Extract displacement vectors from feature point displacement data and extract statistics from background motion feature sets;

[0023] Calculating initial affine transformation parameters according to the displacement vector, adjusting the initial affine transformation parameters based on the statistics, obtaining the affine transformation parameters, and constructing an observation vector according to the affine transformation parameters;

[0024] Based on the observation vector and the Kalman filter, a global motion estimation matrix is ​​output.

[0025] In one embodiment, the step of performing motion compensation on the initial video frame based on the global motion estimation matrix to obtain a stable video data sequence comprises the steps of:

[0026] Selecting the first frame in the initial video data as a reference frame;

[0027] Based on the global motion estimation matrix, calculate the cumulative transformation matrix from the reference frame to any initial video frame;

[0028] The initial video frame is stabilized based on the inverse matrix of the accumulated transformation matrix to obtain a stable video data sequence.

[0029] In one embodiment, the steps of detecting and tracking human key points on a stable video data sequence to obtain an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain motion component signals include the following steps:

[0030] A deep learning model is used to identify a set of human key points in each stable video data sequence, and the detection coordinates of the human key points are obtained; the human key points identified in each stable video data sequence are the same;

[0031] Predicting the predicted coordinates of the key points of the human body in the stable video data sequence of the current frame in the stable video data sequence of the next frame based on the Kalman filter;

[0032] The predicted coordinates are matched with the detected coordinates of the key points of the human body in the stable video data sequence of the next frame using the Hungarian algorithm to obtain the initial motion trajectory sequence of the key points of the human body in the stable video data sequence;

[0033] The initial motion trajectory sequence is decomposed to obtain motion component signals.

[0034] In one embodiment, the step of decomposing the initial motion trajectory sequence to obtain motion component signals includes the steps of:

[0035] Convert the detected coordinates of the key points of the human body from the image coordinate system to the body coordinate system with the hip center as the origin to obtain the converted coordinates;

[0036] The transformed coordinates are transformed from the time series to the frequency domain using Fourier transform to obtain the frequency domain representation of the transformed coordinates;

[0037] The transformed coordinates are transformed using a wavelet transform to obtain a time-frequency representation of the transformed coordinates;

[0038] Decompose the transformed coordinates using empirical mode decomposition to obtain the intrinsic mode function of the transformed coordinates;

[0039] Perform principal component analysis on the initial motion trajectory sequence to obtain the principal components;

[0040] The frequency domain representation, time-frequency representation, intrinsic mode function and principal components are combined to form motion component signals.

[0041] In one embodiment, the steps of using motion component signals to analyze the position differences and time intervals of adjacent key points of a human body, determining a screening factor of the motion component signals, and screening the motion component signals according to the screening factor to obtain abnormal component signals include the following steps:

[0042] Calculate the Euclidean distance between adjacent human key points in each initial video frame, and calculate the mean and standard deviation of the Euclidean distance;

[0043] Calculate the displacement of human key points in consecutive initial video frames, and calculate the mean and standard deviation of the displacement;

[0044] According to the motion component signal, the peak frequency of the main frequency components of the key points of the human body in each initial video frame is identified, and the mean and standard deviation of the peak frequency are calculated;

[0045] According to the motion component signal, the time-frequency energy of the key points of the human body in each initial video frame is calculated, and the mean and standard deviation of the time-frequency energy are calculated;

[0046] According to the motion component signal, the instantaneous amplitude of the key points of the human body in each initial video frame is calculated, and the mean and standard deviation of the instantaneous amplitude are calculated;

[0047] According to the motion component signal, the initial motion trajectory sequence is projected into the principal component space to obtain the projection coefficient, and the mean and standard deviation of the projection coefficient are calculated;

[0048] Constructing screening factors according to Euclidean distance, mean and standard deviation of Euclidean distance, displacement, mean and standard deviation of displacement, peak frequency, mean and standard deviation of peak frequency, time-frequency energy, mean and standard deviation of time-frequency energy, instantaneous amplitude, mean and standard deviation of instantaneous amplitude, projection coefficient and mean and standard deviation of projection coefficient;

[0049] When the screening factor corresponding to the motion component signal is greater than a preset threshold, the motion component signal with the screening factor greater than the preset threshold is an abnormal component signal.

[0050] In one embodiment, the step of extracting features from the abnormal component signal to obtain a weight feature vector comprises the steps of:

[0051] Preprocessing abnormal component signals;

[0052] Input the preprocessed abnormal component signal into the temporal convolutional network and output the feature map;

[0053] The feature map passes through the global average pooling layer to compress the time dimension and obtain the global feature vector;

[0054] The global feature vector is processed by a fully connected layer and outputs a weighted feature vector.

[0055] A computer device comprises a memory and a processor, wherein the memory stores a computer program and the processor implements the steps of the above method when executing the computer program.

[0056] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0057] One of the above technical solutions has the following advantages and beneficial effects:

[0058] The present application comprises the following steps: collecting initial video data for student training, segmenting the initial video data to obtain multiple initial video frames, and calculating the displacement of feature points between adjacent initial video frames to obtain multiple feature point displacement data; extracting background motion features of the initial video frames based on the feature point displacement data to obtain a background motion feature set; performing time series correlation analysis on the feature point displacement data and the background motion feature set to obtain a global motion estimation matrix; performing motion compensation on the initial video frames based on the global motion estimation matrix to obtain a stable video data sequence; performing human body key point detection and tracking on the stable video data sequence to obtain an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain motion component signals; and utilizing the motion component signals to obtain a stable video data sequence. number, analyze the position difference and time interval of adjacent human key points, determine the screening factor of the motion component signal, and screen the motion component signal according to the screening factor to obtain the abnormal component signal; extract the features of the abnormal component signal to obtain the weight feature vector; determine the weight corresponding to the abnormal component signal according to the weight feature vector, the screening factor and the initial motion trajectory sequence; determine the correction weight of the abnormal component signal according to the screening factor and the weight, and correct the initial motion trajectory sequence based on the correction weight to obtain the corrected motion trajectory sequence; compare the corrected motion trajectory sequence with the preset standard action template, calculate the deviation value of the student's human key points and limbs, and generate a comprehensive evaluation report based on the deviation value and the preset scoring rules. This application can obtain accurate feedback on the action in a timely manner by analyzing the students' actions in the initial video data, and efficiently identify the individual differences and specific needs of each student, so as to formulate a personalized training plan, thereby improving the training quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of the flow of the computer vision sports skills assessment method based on signal processing in an embodiment of the present application.

[0060] Figure 2 Schematic diagram of the internal structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] In order to achieve the above object, in one embodiment, Figure 1 As shown, a computer vision sports skill evaluation method based on signal processing is provided, comprising the following steps:

[0063] Step S110, collecting initial video data for student training, segmenting the initial video data to obtain multiple initial video frames, and calculating the displacement of feature points between adjacent initial video frames to obtain multiple feature point displacement data.

[0064] Use a high-speed camera to collect the initial video data for student training. The camera frame rate should be no less than 60fps (frames per second) and the resolution should be at least 1080p (1920x1080 pixels) to ensure image clarity. During the shooting process, the camera should be fixed on a tripod to reduce unnecessary shaking. At the same time, it is necessary to ensure that the shooting environment is well lit to avoid blurry or dark images.

[0065] After acquiring the initial video data, video frame segmentation is performed. This process converts a continuous video stream into a series of still images. Assuming the video frame rate is 60fps, each second of video will be segmented into 60 separate images. This segmentation process can be done with specialized video processing software or with video processing libraries (such as OpenCV) in programming languages ​​(such as Python). Each segmented frame is assigned a unique timestamp, which is very important for accurately locating the specific time when each action occurs during subsequent analysis.

[0066] After the video frame segmentation is completed, a series of initial video data is obtained. This image sequence can be represented by mathematical symbols as {F_1, F_2, ..., F_T}, where F_i represents the i-th frame image and T represents the total number of frames in the video.

[0067] The next step is to calculate the displacement of feature points between adjacent initial video frames. First, feature points need to be detected in each frame. Feature points are points in an image that are easy to identify and track, usually including corners, edges or other significant features of an object. In sports action analysis, these feature points may include key nodes of the student's body (such as wrists, elbows, shoulders, etc.) or specific parts of sports equipment (such as the grip of a racket, the center of the ball, etc.).

[0068] Using the Harris corner detection algorithm or other feature point detection methods, a series of feature points can be detected in each frame of the image. The feature points in the i-th frame are represented as a set P_i={p_i1,p_i2,...,p_iN_i}, where p_ij represents the coordinates of the j-th feature point in the i-th frame, and N_i is the number of feature points detected in the i-th frame.

[0069] After the feature points are detected, it is necessary to track the displacement of these feature points between adjacent frames. Optical flow algorithms, such as the Lucas-Kanade method, can be used to calculate the displacement of feature points between adjacent frames.

[0070] For two adjacent frames F_i and F_(i+1), calculate the displacement vector of each feature point. Let d_ij represent the displacement from the jth feature point in the i-th frame to the corresponding feature point in the i+1-th frame. Then, the feature point displacement data can be obtained:

[0071] D_i={d_i1,d_i2,...,d_iM_i}

[0072] Where d_ij = p_(i+1)j-p_ij, Mi is the number of feature points successfully matched between the i-th frame and the i+1-th frame. Note that Mi may be less than or equal to the smaller value of N_i and N_(i+1) because not all feature points can be successfully matched in adjacent frames.

[0073] In this way, for the entire video sequence, a series of feature point displacement data D_i (i=1,2,...,T-1) can be obtained. Each D_i contains the displacement vectors of the successfully matched feature points from the i-th frame to the i+1-th frame. These displacement vectors constitute the feature point displacement data, which describes the movement of all successfully matched feature points from one frame to the next.

[0074] Step S120, extracting background motion features of the initial video frame according to the feature point displacement data to obtain a background motion feature set.

[0075] In one example, the steps of extracting background motion features of an initial video frame based on feature point displacement data and obtaining a background motion feature set include the following steps: analyzing feature point displacement data based on a random sampling consistency algorithm to obtain a background motion feature model between each pair of adjacent initial video frames; detecting a group of background feature points from the initial video frame; predicting the predicted position of the background feature points of the previous initial video frame between the corresponding adjacent initial video frames in the next initial video frame based on the background motion feature model, and calculating the error between the predicted position and the actual position of the background feature points of the next initial video frame; defining a background motion feature vector corresponding to the initial video frame based on the error, and constructing a background motion feature set based on the background motion feature vector.

[0076] First, feature points that may belong to the background are separated from the feature point displacement data D_i. This can be achieved by analyzing the motion consistency of the feature points. Usually, the motion of background feature points will show a more consistent pattern, while student-related feature points will have greater changes and inconsistencies. For each pair of adjacent frames F_i and F_(i+1), the RANSAC (Random Sample Consensus) algorithm is used to estimate the background motion model. The RANSAC algorithm is able to robustly fit the model in data containing a large number of outliers. In this scenario, student-related feature points can be regarded as outliers, while background feature points are regarded as inliers.

[0077] The RANSAC algorithm works by randomly selecting the minimum number of sample points from D_i to fit the initial model. For example, for an affine transformation model, at least three non-collinear points are required. The algorithm then calculates the distance from all other data points to this initial model, and marks the points with a distance less than a predetermined threshold t as inliers. If the number of inliers exceeds another predetermined threshold (usually a certain proportion of the total number of data points), the model is considered to be a good candidate model, and all inliers are used to re-estimate the model parameters to obtain a more accurate model. This process is repeated many times (for example, 1000 times), and the model with the most inliers is finally selected as the best background motion model.

[0078] In this process, affine transformation is used as the background motion model. Affine transformation can describe transformations such as translation, rotation, scaling and shearing, which is sufficient for most cases. Affine transformation can be represented by a 2x3 matrix A: [x',y']=[x,y,1]*A, where (x,y) is the original coordinate and (x',y') is the transformed coordinate. This transformation matrix A contains six parameters and can fully describe the affine transformation on a two-dimensional plane.

[0079] Through the RANSAC algorithm, the background motion model A_i between each pair of adjacent initial video frames is obtained. In this way, the background motion model set M can be defined as: M={A_1,A_2,...,A_(T-1)}, where A_i represents the background motion model from the i-th frame to the i+1-th frame. This set M describes the motion changes of the background in the entire video sequence.

[0080] Next, in order to extract more accurate background motion features, these background motion models need to be applied to each initial video frame. For each initial video frame F_i, a new set of background feature points is detected using the Harris corner detector or other feature point detection algorithms (such as SIFT, SURF, etc.). These background feature points may not be exactly the same as the feature points used in step 1, because we are now focusing on background features. Let this set of feature points be Q_i={q_i1,q_i2,...,q_iK_i}, where K_i is the number of background feature points detected in the i-th frame. When selecting these background feature points, those areas identified as internal points in the RANSAC process can be given priority, because these areas are more likely to belong to the background.

[0081] Using the background motion model A_i, predict the position of these background feature points in the next frame: q'_ij=A_i*[q_ij,1]^T (where j=1,2,...,K_i). This step actually maps the background feature points of the current initial video frame to the predicted position in the next initial video frame through affine transformation. Then, calculate the error between these predicted positions and the actual positions. This error can be measured by the Euclidean distance: error_ij=||q'_ij-q_(i+1)j||, where q_(i+1)j is the position of the corresponding feature point actually detected in the next frame.

[0082] These error values ​​provide important information that can be used to evaluate the accuracy of the background motion model and further optimize the model. If the error of some feature points is particularly large, it may mean that these points do not belong to the background, or that the local background has undergone complex changes (for example, the swaying of leaves).

[0083] Based on these errors, the background motion feature vector b_i of each initial video frame is defined. This vector can contain multiple statistics, such as:

[0084] -mean_error: the average value of all feature point errors

[0085] -std_error: The standard deviation of the error, which reflects the degree of dispersion of the error distribution

[0086] -max_error: maximum error, which can help identify outliers

[0087] -median_error: median error, insensitive to outliers

[0088] -inlier_ratio: The ratio of inliers (points with errors less than a certain threshold) to the total feature points

[0089] Formally, it can be expressed as: b_i=[mean_error,std_error,max_error,median_error,inlier_ratio,...]. This feature vector b_i fully describes the characteristics of the background motion between the i-th frame and the i+1-th frame.

[0090] Through this process, we finally get the background motion feature set B={b_i|i=1,2,...,T}, where each b_i is the background motion feature vector corresponding to the i-th frame.

[0091] Step S130 , performing temporal correlation analysis on the feature point displacement data and the background motion feature set to obtain a global motion estimation matrix.

[0092] In one example, a step of performing temporal correlation analysis on feature point displacement data and background motion feature set to obtain a global motion estimation matrix includes the steps of: extracting a displacement vector from the feature point displacement data and extracting a statistic from the background motion feature set; calculating initial affine transformation parameters according to the displacement vector, adjusting the initial affine transformation parameters based on the statistic, obtaining the affine transformation parameters, and constructing an observation vector according to the affine transformation parameters; and outputting a global motion estimation matrix based on the observation vector and a Kalman filter.

[0093] Considering the continuity and smoothness of motion, this step uses a Kalman filter to build a temporal model to describe the motion changes in the initial video frame. The Kalman filter is a recursive estimator that can optimally estimate the state of a dynamic system in the presence of measurement noise. In this scenario, the system state is defined as a vector containing global motion parameters. Assuming that an affine transformation is used to describe global motion, the state vector x_k can be expressed as: x_k=[a11,a12,a21,a22,tx,ty,da11,da12,da21,da22,dtx,dty]^T, where a11,a12,a21,a22,tx,ty are the six parameters of the affine transformation matrix, representing rotation, scaling, shearing, and translation, respectively, and da11,da12,da21,da22,dtx,dty are the rates of change of these parameters, which help the model better capture the dynamic characteristics of motion.

[0094] The prediction step of the Kalman filter can be expressed as: x_k|k-1=F*x_k-1|k-1 and P_k|k-1=F*P_k-1|k-1*F^T+Q, where x_k|k-1 is the prior state estimate at time k, F is the state transfer matrix, P_k|k-1 is the prior error covariance matrix, and Q is the process noise covariance matrix. The state transfer matrix F is designed as: F=[I_6,Δt*I_6;0_6,I_6], where I_6 is a 6x6 identity matrix, Δt is the time interval between adjacent frames, and 0_6 is a 6x6 zero matrix. This design reflects the relationship between motion parameters and their rates of change, allowing the model to capture the effects of velocity and acceleration.

[0095] The measurement update step is a key step in the Kalman filter, which makes comprehensive use of the feature point displacement data Di and the background motion features b to correct the prediction. The core of this process is to construct the observation vector zk, which requires a clever fusion of these two sources of information. First, a representative displacement vector is extracted from the feature point displacement data Di. This is usually achieved by calculating the median or weighted average of all feature point displacements. For example, the weighted average displacement can be calculated using the following formula: Δp_avg=Σ(w_i*Δp_i) / Σw_i, where Δp_i is the displacement of the i-th feature point, and w_i is the corresponding weight, which can be determined based on the tracking quality or other reliability indicators of the feature point.

[0096] At the same time, statistics in the background motion features b_i, such as mean_error (mean error) and inlier_ratio (inlier ratio), are used to evaluate the reliability of the background motion model. These indicators reflect the accuracy and consistency of background motion estimation. Lower mean_error and higher inlier_ratio indicate that the background motion model is very reliable.

[0097] Based on this information, construct the observation vector z_k=[a11_obs,a12_obs,a21_obs,a22_obs,tx_obs,ty_obs]^T. This observation vector represents the six parameters of the affine transformation. The construction process first estimates the initial affine transformation parameters using the displacement data in D_i. This can be achieved by solving the least squares problem: minimizeΣ||p_i'-A*p_i||^2, where p_i and p_i' are the positions of the corresponding feature points in two adjacent frames, respectively, and A is the affine transformation matrix. Then, these initial estimates are adjusted according to the reliability indicators in b_i. The degree of adjustment depends on the reliability of the background motion model. The following weighted average method can be used: a_ij_obs=λ*a_ij_D+(1-λ)*a_ij_B, where a_ij_D is the parameter estimated from D_i, a_ij_B is the parameter obtained from the background motion model, and λ is the weight factor, which can be determined based on inlier_ratio and mean_error. For example: λ=(1-inlier_ratio)*(mean_error / threshold_error), where threshold_error is a predefined error threshold.

[0098] In this way, when the background motion model is very reliable (high inlier_ratio and low mean_error), the observations will rely more on the background motion model; otherwise, they will rely more on the direct displacement data in D_i. This trade-off mechanism ensures reliable observation estimates in different scenarios.

[0099] With the observation vector z_k, the update step of the Kalman filter is performed according to the standard formula:

[0100] 1. Calculate the measurement residual: y_k=z_k-H*x_k|k-1

[0101] 2. Calculate the residual covariance: S_k=H*P_k|k-1*H^T+R

[0102] 3. Calculate Kalman gain: K_k=P_k|k-1*H^T*S_k^-1

[0103] 4. Update state estimate: x_k|k=x_k|k-1+K_k*y_k

[0104] 5. Update error covariance: P_k|k=(I-K_k*H)*P_k|k-1

[0105] Here, the observation matrix H=[I_6,0_6] means that only the motion parameters are directly observed, but not their rate of change. The observation noise covariance matrix R can be dynamically adjusted according to the reliability of D_i and b_i, for example: R=diag([σ_a^2,σ_a^2,σ_a^2,σ_a^2,σ_t^2,σ_t^2]*(1+β*(1-inlier_ratio))), where σ_a and σ_t are the basic noise levels of rotation and translation parameters, respectively, and β is an adjustment factor. This method of dynamically adjusting R enables the filter to adaptively adjust its trust in the observation according to the reliability of the observation.

[0106] By iteratively applying the prediction and update steps of the Kalman filter, a global motion estimate is obtained for each initial video frame. These estimates form a time series that describes the evolution of motion throughout the video. In order to obtain the final global motion estimate matrix G, the output of the Kalman filter needs to be converted into a suitable form. For each frame k, the affine transformation parameters are extracted from the state estimate x_k|k to construct a 3x3 matrix:

[0107] G_k=[a11_k,a12_k,tx_k;a21_k,a22_k,ty_k;0,0,1]

[0108] This matrix G_k completely describes the global motion transformation from the kth frame to the k+1th frame. It contains all possible two-dimensional transformation information such as rotation, scaling, shearing and translation. In this way, we finally get a series of global motion estimation matrices G={G_1,G_2,...,G_T}, where T is the total number of frames in the video.

[0109] This global motion estimation matrix sequence G not only contains the motion information between each pair of adjacent frames, but also implies the motion trajectory of the entire video sequence. By continuously applying these transformation matrices, the motion path of any feature point or object in the entire video can be reconstructed.

[0110] Step S140 , performing motion compensation on the initial video frame based on the global motion estimation matrix to obtain a stable video data sequence.

[0111] In one example, based on a global motion estimation matrix, motion compensation is performed on an initial video frame to obtain a stable video data sequence, including the steps of: selecting the first frame in the initial video data as a reference frame; based on the global motion estimation matrix, calculating a cumulative transformation matrix from the reference frame to any initial video frame; and stabilizing the initial video frame based on an inverse matrix of the cumulative transformation matrix to obtain a stable video data sequence.

[0112] First, during the motion compensation process, a suitable reference frame needs to be selected. The first frame of the video data is usually selected as the reference frame because this provides a natural starting point and simplifies the subsequent calculation process. This selection is based on the assumption that the beginning of the video usually contains the complete initial pose or action. However, other frames can also be selected as references depending on the specific analysis requirements. For example, if the first frame is of poor quality due to motion blur or occlusion, a clear and representative frame can be selected as a reference. When selecting a reference frame, its position in the entire sequence must also be considered to minimize the cumulative error. This is because small errors may accumulate and amplify over time, resulting in poor correction effects in subsequent frames.

[0113] Assuming that the first frame is chosen as the reference frame, the next step is to calculate the cumulative transformation matrix A_k from the first frame to any initial video frame k. This matrix captures all the motion information from the reference frame to the current frame. The process of calculating A_k is achieved through continuous multiplication: A_k=G_1*G_2*...*G_(k-1), where G_i represents the global motion estimation matrix from the i-th frame to the i+1-th frame. This accumulation method ensures that the transformation of each frame takes into account all intermediate transformations starting from the reference frame. Specifically, for the second frame, A_2 is equal to G_1; for the third frame, A_3=G_1*G_2; and so on. It is worth noting that A_1 (the transformation matrix of the first frame) is the identity matrix, because the first frame is the reference frame itself and does not require any transformation.

[0114] After calculating A_k, the next step is to apply an inverse transform to each initial video frame to eliminate the effects of global motion. This process can be expressed as: F_k_stabilized=A_k^(-1)*F_k, where F_k is the original k-th frame, F_k_stabilized is the stabilized k-th frame, and A_k^(-1) is the inverse matrix of A_k. Here, F_k and F_k_stabilized both represent images and can be viewed as two-dimensional matrices (for grayscale images) or three-dimensional tensors (for color images). This representation method is consistent with {F_1,F_2,...,F_T} in step 1, where F_i represents the i-th frame image and T represents the total number of frames in the video.

[0115] The physical meaning of this transformation is to "pull back" each initial video frame to the perspective of the reference frame. Specifically, A_k describes how to transform from the reference frame to the current frame, and A_k^(-1) describes how to transform from the current frame back to the reference frame. Applying A_k^(-1) to F_k is equivalent to undoing all the accumulated motion from the reference frame to the current frame, so that the current frame is aligned with the reference frame in spatial position. This process can be likened to correcting the perspective of a photo, just like adjusting photos taken from different angles to the same perspective.

[0116] In practice, this transformation is achieved through image deformation technology. The transformation matrix A_k^(-1) defines the position of each pixel in the original image in the new image. The specific steps are as follows: First, create a blank image of the same size as the original image as the target image F_k_stabilized. Then, for each pixel position (x',y') of the target image F_k_stabilized, use the inverse transformation matrix A_k^(-1) to calculate its corresponding position (x,y) in the original image F_k. This calculation can be expressed as: [x,y,1]^T=A_k^(-1)*[x',y',1]^T. Since the calculated (x,y) is usually not an integer coordinate, an interpolation method is required to determine the pixel value at that position. Common interpolation methods include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. Among them, bilinear interpolation is a more commonly used method, which uses the weighted average of the four integer coordinate pixel values ​​around (x,y) to determine the pixel value at that position. This method achieves a good balance between computational efficiency and image quality.

[0117] The specific calculation process of bilinear interpolation is as follows: Assume that the calculated non-integer coordinate is (x, y), and the four integer coordinate pixels around it are (x1, y1), (x1, y2), (x2, y1) and (x2, y2), where x1=floor(x), x2=ceil(x), y1=floor(y), y2=ceil(y). Let f(x, y) represent the pixel value at the (x, y) position, then the calculation formula of bilinear interpolation is:

[0118] f(x,y)=f(x1,y1)*(x2-x)*(y2-y)+f(x2,y1)*(x-x1)*(y2-y)+f(x1,y2)*(x2-x)*(y-y1)+f(x2,y2)*(x-x1)*(y-y1)

[0119] This formula actually performs linear interpolation in the x direction first, then performs linear interpolation in the y direction, and finally obtains the interpolation result at the (x, y) position.

[0120] After the interpolation is completed, the obtained pixel value is assigned to the pixel at the position (x', y') of F_k_stabilized. This process is repeated until all pixels of F_k_stabilized are assigned. This process is performed for each frame in the video sequence (except the reference frame), and finally a stable video data sequence {F_1_stabilized, F_2_stabilized, ..., F_T_stabilized} is obtained.

[0121] Step S150 , performing human body key point detection and tracking on the stable video data sequence to obtain an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain motion component signals.

[0122] In one example, a stable video data sequence is subjected to human key point detection and tracking to obtain an initial motion trajectory sequence, and the initial motion trajectory sequence is decomposed to obtain a motion component signal, including the steps of: using a deep learning model to respectively identify a group of human key points in each stable video data sequence, and obtaining the detection coordinates of the human key points; the human key points identified in each stable video data sequence are the same; based on a Kalman filter, the predicted coordinates of the human key points in the stable video data sequence of the current frame in the stable video data sequence of the next frame are predicted; using a Hungarian algorithm to match the predicted coordinates with the detection coordinates of the human key points in the stable video data sequence of the next frame to obtain an initial motion trajectory sequence of the human key points in the stable video data sequence; and decomposing the initial motion trajectory sequence to obtain a motion component signal.

[0123] For each frame in the stabilized video data sequence {F_1_stabilized, F_2_stabilized, ..., F_T_stabilized}, human keypoint detection is performed. This process uses the deep learning model OpenPose to identify and locate the key points of the human body. These key points include the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. The OpenPose model uses a multi-stage convolutional neural network to first detect heat maps of body parts and then connect these parts into a complete skeleton through part association fields (PAFs). For each frame F_k_stabilized, the model outputs a set of keypoint coordinates: P_k={p_k1, p_k2, ..., p_kN}, where p_ki=(x_ki, y_ki) represents the coordinates of the i-th keypoint in the k-th frame, and N is the total number of keypoints detected. Typically, the value of N is between 15 and 25, depending on the specific model used. For the entire video sequence, a time series of keypoint coordinates is obtained: P={P_1, P_2, ..., P_T}.

[0124] Next, the detected key points of the human body are tracked. In order to obtain a coherent motion trajectory, it is necessary to establish a correspondence between the key points of the human body between frames. Here, a motion prediction method based on the Kalman filter is adopted. For each key point, a motion model is established using the Kalman filter. The Kalman filter is a recursive estimator that optimally estimates the state of a dynamic system in the presence of measurement noise. In this application, the state vector of the Kalman filter includes the position and velocity of the key point, usually expressed as x=[x,y,vx,vy]^T. The prediction step uses a constant velocity model: x_k|k-1=F*x_k-1|k-1, where F is the state transfer matrix. The update step uses the detected key point positions as observations to update the state estimate of the filter.

[0125] In each frame, the key point positions of the current frame are first predicted using the state of the previous frame, and then the predicted coordinates are matched with the actual detection coordinates. The matching process uses the Hungarian algorithm, also known as the Kuhn-Munkres algorithm, which is a method for finding the optimal match in a many-to-many assignment problem. Here, the goal of the algorithm is to minimize the total distance between the predicted position and the detected position. The Hungarian algorithm constructs a cost matrix in which each element represents the distance from a predicted position to a detected position, and then finds the matching solution with the minimum total cost in polynomial time.

[0126] This tracking process is performed independently for each human key point, resulting in a continuous trajectory for each human key point throughout the video sequence. In this way, even if the detection of a key point in some frames fails or is inaccurate, the continuity of the trajectory can be maintained through prediction. At the same time, the use of the Kalman filter can also smooth the trajectory and reduce the impact of detection noise.

[0127] By tracking, we get the initial motion trajectory sequence of each key point in the entire video sequence. It can be expressed as Γ={T_1,T_2,...,T_N}, where T_i={p_1i,p_2i,...,p_Ti} represents the trajectory of the i-th human key point in all frames. Each trajectory T_i is a two-dimensional time series that records the changes in the x and y coordinates of a specific key point over the duration of the video.

[0128] In one example, in one of the embodiments, the step of decomposing the initial motion trajectory sequence to obtain the motion component signal includes the steps of: converting the detected coordinates of the key points of the human body from the image coordinate system to the body coordinate system with the hip center as the origin to obtain the converted coordinates; using Fourier transform to convert the converted coordinates from the time series to the frequency domain to obtain the frequency domain representation of the converted coordinates; using wavelet transform to transform the converted coordinates to obtain the time-frequency representation of the converted coordinates; using empirical mode decomposition to decompose the converted coordinates to obtain the intrinsic mode function of the converted coordinates; performing principal component analysis on the initial motion trajectory sequence to obtain the principal components; and combining the frequency domain representation, time-frequency representation, intrinsic mode function and principal components into a motion component signal.

[0129] The obtained initial motion trajectory sequence is decomposed to obtain motion component signals. First, the detected coordinates of the key points of the human body are converted from the image coordinate system to the body coordinate system with the center of the hip as the origin. This conversion process is crucial to eliminate the influence of overall displacement and focus on the relative movement of body parts. For each frame, the midpoint of the left and right hip key points is calculated as the origin of the new coordinate system. Then, the direction of the line connecting the left and right hips is used to define the x-axis of the new coordinate system, and the direction perpendicular to the x-axis is defined as the y-axis. The coordinate system defined in this way can change with the rotation and movement of the human body, and always maintain consistency with the body posture. The coordinates of each key point are transformed as follows:

[0130] x_new=(x_old-x_origin)*cos(θ)+(y_old-y_origin)*sin(θ)

[0131] y_new=-(x_old-x_origin)*sin(θ)+(y_old-y_origin)*cos(θ)

[0132] Among them, (x_old, y_old) is the original coordinate, (x_origin, y_origin) is the origin of the new coordinate system (hip center), and θ is the rotation angle of the x-axis relative to the image coordinate system. This transformation is actually a translation plus rotation process, which ensures that the positions of all key points are described relative to the center of the body.

[0133] Next, the x_new and y_new coordinate time series of each key point of the human body are decomposed separately. Common decomposition methods include Fourier transform, wavelet transform and empirical mode decomposition (EMD). Fourier transform converts the time series to the frequency domain, and the main periodic components can be identified. For each key point trajectory T_i, its spectrum representation can be obtained: X_i(f)=FFT(x_i(t)) and Y_i(f)=FFT(y_i(t)), where x_i(t) and y_i(t) are the time series of x and y coordinates in T_i, respectively. Wavelet transform provides time-frequency representation, which is suitable for analyzing non-stationary signals and can be expressed as: W_xi(a,b)=∫x_i(t)*ψ((tb) / a)dt and W_yi(a,b)=∫y_i(t)*ψ((tb) / a)dt, where ψ(t) is the wavelet function, a is the scale parameter, and b is the translation parameter. Empirical mode decomposition decomposes the signal into a series of intrinsic mode functions (IMFs): x_i(t)=∑IMF_xij(t)+r_xi(t) and y_i(t)=∑IMF_yij(t)+r_yi(t), where IMF_xij and IMF_yij are the intrinsic mode functions obtained by decomposition, and r_xi and r_yi are the residuals.

[0134] In order to extract the main motion patterns, principal component analysis (PCA) can be performed on the trajectories of all key points of the human body. First, construct a data matrix X, where each row represents a time point and each column represents the coordinates of a key point. Then calculate the covariance matrix C=X^T*X, and perform eigenvalue decomposition on C: C=V*D*V^T. Select the eigenvectors corresponding to the largest eigenvalues ​​as the principal components. The principal components can be expressed as P=X*V, where V is the selected eigenvector matrix.

[0135] Finally, through these decompositions and transformations, a series of motion component signals are obtained, including frequency domain representation X_i(f) and Y_i(f), time-frequency representation W_xi(a,b) and W_yi(a,b), intrinsic mode functions IMF_xij(t) and IMF_yij(t), and principal component P.

[0136] Step S160, using the motion component signal, analyzing the position difference and time interval of adjacent human body key points, determining the screening factor of the motion component signal, and screening the motion component signal according to the screening factor to obtain an abnormal component signal.

[0137] In one example, a motion component signal is used to analyze the position difference and time interval of adjacent human key points, determine the screening factor of the motion component signal, and screen the motion component signal according to the screening factor to obtain the abnormal component signal, including the steps of: calculating the Euclidean distance of adjacent human key points in each initial video frame, and calculating the mean and standard deviation of the Euclidean distance; calculating the displacement of the human key points in consecutive initial video frames, and calculating the mean and standard deviation of the displacement; identifying the peak frequency of the main frequency components of the human key points in each initial video frame according to the motion component signal, and calculating the mean and standard deviation of the peak frequency; calculating the time-frequency energy of the human key points in each initial video frame according to the motion component signal, and calculating the mean and standard deviation of the time-frequency energy ; According to the motion component signal, the instantaneous amplitude of the key points of the human body in each initial video frame is calculated, and the mean and standard deviation of the instantaneous amplitude are calculated; according to the motion component signal, the initial motion trajectory sequence is projected into the principal component space to obtain the projection coefficient, and the mean and standard deviation of the projection coefficient are calculated; according to the Euclidean distance, the mean and standard deviation of the Euclidean distance, the displacement, the mean and standard deviation of the displacement, the peak frequency, the mean and standard deviation of the peak frequency, the time-frequency energy, the mean and standard deviation of the time-frequency energy, the instantaneous amplitude, the mean and standard deviation of the instantaneous amplitude, the projection coefficient and the mean and standard deviation of the projection coefficient, a screening factor is constructed; when the screening factor corresponding to the motion component signal is greater than a preset threshold, the motion component signal whose screening factor is greater than the preset threshold is an abnormal component signal.

[0138] For each pair of adjacent human key points i and j, calculate their Euclidean distance in each frame k:

[0139] d_ij(k)=√((x_ki-x_kj)^2+(y_ki-y_kj)^2)

[0140] Where x_ki and y_ki represent the x and y coordinates of key point i in the kth frame, respectively, and x_kj and y_kj represent the coordinates of key point j. This distance sequence d_ij reflects the change in the relative position between two key points. Calculate the statistical features of this distance sequence, including the mean μ_ij and standard deviation σ_ij:

[0141] μ_ij=(1 / T)*Σ(d_ij(k))(k from 1 to T)

[0142] σ_ij=√((1 / T)*Σ((d_ij(k)-μ_ij)^2))(k from 1 to T)

[0143] Where T is the total number of frames in the video sequence. These statistics are used to evaluate the stability and consistency of the relative motion between key points.

[0144] Next, we analyze the time interval features. For each key point i, we calculate its displacement between consecutive frames:

[0145] Δs_i(k)=√((x_ki-x_k-1,i)^2+(y_ki-y_k-1,i)^2)

[0146] Similarly, calculate the statistical characteristics of this displacement sequence, including the mean μ_Δs_i and the standard deviation σ_Δs_i:

[0147] μ_Δs_i=(1 / (T-1))*Σ(Δs_i(k))(k from 2 to T)

[0148] σ_Δs_i=√((1 / (T-1))*Σ((Δs_i(k)-μ_Δs_i)^2))(k from 2 to T)

[0149] μ_Δs_i reflects the average movement speed of key point i, while σ_Δs_i describes the degree of speed change.

[0150] Using the frequency domain representation X_i(f) and Y_i(f), the main frequency components of each key point motion can be identified. For each key point i and each frame k, find the peak frequency in the energy spectrum:

[0151] f_i(k)=argmax_f(|X_i(f,k)|^2+|Y_i(f,k)|^2)

[0152] Then calculate the mean μ_f_i and standard deviation σ_f_i of f_i(k) in the entire sequence:

[0153] μ_f_i=(1 / T)*Σ(f_i(k))(k ranges from 1 to T)

[0154] σ_f_i=√((1 / T)*Σ((f_i(k)-μ_f_i)^2))(k from 1 to T)

[0155] These frequency features can reflect the periodicity and rhythmic changes of the movement. A lower μ_fi may indicate a slow movement, while a higher μ_fi may indicate a fast or repetitive action.

[0156] The time-frequency representation W_xi(a,b) and W_yi(a,b) provide more detailed time-varying frequency information. For each time instant k, the sum of the time-frequency energy is calculated:

[0157] E_i(k)=Σ_a(|W_xi(a,k)|^2+|W_yi(a,k)|^2)

[0158] Then calculate the mean μ_E_i and standard deviation σ_E_i of E_i(k):

[0159] μ_E_i=(1 / T)*Σ(E_i(k))(k from 1 to T)

[0160] σ_E_i=√((1 / T)*Σ((E_i(k)-μ_E_i)^2))(k from 1 to T)

[0161] These statistics can capture instantaneous changes in motion and changes in energy distribution. Higher E_i(k) values ​​may correspond to critical moments or phases of higher intensity in the motion.

[0162] For the intrinsic mode functions IMF_xij(t) and IMF_yij(t), calculate the instantaneous amplitude of each IMF:

[0163] A_ij(k)=√(IMF_xij(k)^2+IMF_yij(k)^2)

[0164] Calculate the mean μ_A_ij and standard deviation σ_A_ij for A_ij(k):

[0165] μ_A_ij=(1 / T)*Σ(A_ij(k))(k from 1 to T)

[0166] σ_A_ij=√((1 / T)*Σ((A_ij(k)-μ_A_ij)^2))(k from 1 to T)

[0167] These features can reflect changes at different scales in motion. Lower-order IMFs correspond to faster oscillations and may reflect subtle motion adjustments, while higher-order IMFs may correspond to overall motion trends.

[0168] Using the result P of principal component analysis, the original trajectory is projected into the principal component space. Assuming that the first m principal components are retained, then for each time k, m projection coefficients can be obtained:

[0169] c_ij(k)=[c_i1(k),c_i2(k),...,c_im(k)](j=1,2,...,m)

[0170] Here, c_ij(k) represents the principal component projection coefficient vector of the i-th key point in the k-th frame, and j represents different principal components.

[0171] Compute the mean and standard deviation for each coefficient:

[0172] μ_c_ij=(1 / T)*Σ(c_ij(k)) (k from 1 to T)

[0173] σ_c_ij=√((1 / T)*Σ((c_ij(k)-μ_c_ij)^2))(k from 1 to T)

[0174] Principal component analysis can reveal the main variation patterns and overall structure of the movement. The first principal component usually captures the largest variation, while subsequent principal components reflect more subtle variation patterns.

[0175] Based on all the above statistics, a comprehensive screening factor F_i(k) is defined. For each key point i and each moment k, the standardized scores of the following indicators are calculated:

[0176] Z_d_i(k)=(d_ij(k)-μ_ij) / σ_ij (j is the adjacent key point of i)

[0177] Z_Δs_i(k)=(Δs_i(k)-μ_Δs_i) / σ_Δs_i

[0178] Z_f_i(k)=(f_i(k)-μ_f_i) / σ_f_i

[0179] Z_E_i(k)=(E_i(k)-μ_E_i) / σ_E_i

[0180] Z_A_i(k)=max_j(|A_ij(k)-μ_A_ij| / σ_A_ij)

[0181] Z_c_i(k)=max_j(|c_ij(k)-μ_c_ij| / σ_c_ij)

[0182] Z_d_i(k): This represents the normalized score of the distance between the ith human keypoint and its neighboring human keypoint j in the kth frame.

[0183] Z_Δs_i(k): This represents the normalized score of the displacement of the i-th human keypoint at the k-th frame (relative to the previous frame).

[0184] Z_f_i(k): This represents the normalized score of the main frequency component of the i-th human keypoint in the k-th frame.

[0185] Z_E_i(k): This represents the normalized score of the time-frequency energy of the i-th human keypoint in the k-th frame.

[0186] Z_A_i(k): This represents the maximum normalized score of the intrinsic mode function (IMF) amplitude of the i-th human keypoint at the k-th frame.

[0187] Z_c_i(k): This represents the maximum normalized score of the principal component projection coefficients of the i-th human keypoint at the k-th frame.

[0188] Then, the comprehensive screening factor F_i(k) is defined as the maximum absolute value of these standardized scores:

[0189] F_i(k)=max(|Z_d_i(k)|,|Z_Δs_i(k)|,|Z_f_i(k)|,|Z_E_i(k)|,|Z_A_i(k)|,|Z_c_i(k)|)

[0190] This comprehensive filtering factor F_i(k) takes into account all signal representations and is able to capture various significant changes in motion. Larger F_i(k) values ​​indicate significant deviations in a certain dimension, which may correspond to important moments or special patterns in the motion.

[0191] Next, the motion component signal is filtered according to this filtering factor F_i(k) to obtain the abnormal component signal. A threshold τ (for example, τ=3) is set. When F_i(k)>τ, it is considered that the motion of key point i in the kth frame is abnormal.

[0192] For the moment k and key point i identified as abnormal, the abnormal component signal is defined as a multidimensional vector, which contains the local information represented by various signals:

[0193] A_i(k)={

[0194] Original trajectory: {(x_i(kw),y_i(kw)),...,(x_i(k),y_i(k)),...,(x_i(k+w),y_i(k+w))},

[0195] Frequency domain representation: {X_i(f,kw:k+w),Y_i(f,kw:k+w)},

[0196] Time-frequency representation: {W_xi(a,kw:k+w),W_yi(a,kw:k+w)},

[0197] IMF:{IMF_xij(kw:k+w),IMF_yij(kw:k+w)} for all j,

[0198] Principal component projection: {c_ij(kw:k+w)} for all j,

[0199] Abnormal dimensions: argmax(|Z_d_i(k)|,|Z_Δs_i(k)|,|Z_f_i(k)|,|Z_E_i(k)|,|Z_A_i(k)|,

[0200] |Z_c_i(k)|),

[0201] F_i(k) value: F_i(k)

[0202] }

[0203] Where w is a small time window (for example, w=5). This definition allows the abnormal component signal to contain information in multiple dimensions, fully reflecting the motion characteristics at the time when the abnormality occurs.

[0204] Combine all detected abnormal component signals into one set:

[0205] A={A_1,A_2,...,A_M}

[0206] Where M is the number of detected anomalies. Each A_m (m=1,2,...,M) represents a detected anomaly component signal, which contains all the information defined above.

[0207] Step S170, extracting features from the abnormal component signal to obtain a weighted feature vector; determining a local weight corresponding to the abnormal component signal according to the weighted feature vector, the screening factor and the initial motion trajectory sequence.

[0208] In one example, the step of extracting features from an abnormal component signal to obtain a weighted feature vector includes the following steps: preprocessing the abnormal component signal; inputting the preprocessed abnormal component signal into a temporal convolutional network, and outputting a feature map through a global average pooling layer to compress the time dimension and obtain a global feature vector; processing the global feature vector through a fully connected layer and outputting a weighted feature vector.

[0209] This process adopts a deep learning-based approach, using a pre-trained temporal convolutional network (TCN) as a feature extractor to efficiently extract features and calculate weights.

[0210] First, the abnormal component signal A_m is preprocessed. The preprocessing includes length normalization and amplitude normalization. Length normalization resamples the signal to a fixed length L, usually 256 time steps, and uses a linear interpolation method to ensure that all signals have the same number of time steps. For each target time step i (i from 0 to L-1), the corresponding original signal index is j=i*(N-1) / (L-1), where N is the original signal length. If j is not an integer, linear interpolation is performed between the integer part and the fractional part of j.

[0211] Amplitude normalization maps the amplitude value of a signal to the range [-1,1]. This step first finds the maximum absolute value of the signal max_abs, and then divides all data points by this maximum value. For signal x, the normalized signal x_norm=x / max_abs. This process ensures the comparability of the amplitudes between different signals while maintaining the relative strength relationship of the signals.

[0212] The preprocessed signal is input into TCN. The structure of TCN consists of an initial one-dimensional convolution layer followed by multiple TCN blocks. The kernel size of the initial convolution layer is 7 and the number of output channels is 64, which is used to preliminarily extract local features. Each TCN block contains two sub-blocks, each of which consists of a one-dimensional dilated convolution layer, a weight normalization layer, a ReLU activation function, and a Dropout layer.

[0213] Dilated convolution is a core component of TCN, which increases the receptive field by inserting "holes" in the standard convolution kernel. The dilation rate of the first TCN block is 1, and the dilation rate of each subsequent block is twice the previous one. This design makes the receptive field of the network grow exponentially with depth. For a network of n TCN blocks, the receptive field size of the last block is 2^(n+1)-1 time steps.

[0214] In each TCN block, the signal first passes through a one-dimensional dilated convolution. The output of the convolution is normalized by a weight normalization layer, and then nonlinearity is introduced by a ReLU activation function. The ReLU function is defined as f(x)=max(0,x), which maintains a linear mapping in the positive range and maps all negative values ​​to zero. Finally, it passes through a Dropout layer (the dropout rate is set to 0.2) to prevent overfitting.

[0215] The output of each TCN block is also residually connected to its input, which can be formally expressed as Output = F (Input) + Input, where F represents a series of operations of the sub-block. This connection method allows the network to learn the residual mapping between input and output, which helps the back propagation of gradients and the training of deep networks.

[0216] After passing through all TCN blocks, the feature map obtained passes through the global average pooling layer to compress the time dimension and obtain a global feature vector. The global average pooling operation takes the average value of each channel in the time dimension, which can greatly reduce the number of parameters while maintaining time invariance. This operation can be seen as a summary of the entire time series, which extracts the average activation intensity of each feature channel over the entire sequence.

[0217] Finally, this feature vector passes through a fully connected layer to output the final weighted feature vector F_m, whose dimension is set to 128. The role of the fully connected layer is to map the pooled features to the feature space of the specified dimension, increasing the nonlinear expression ability of the model. This layer can learn complex combinations between different features to generate a highly abstract and information-rich feature representation. Setting the output dimension to 128 is a balance between feature expression ability and computational efficiency. It provides enough dimensions to capture complex motion patterns without causing excessive computational burden on subsequent processing.

[0218] Next, calculate the local weight of the abnormal component signal. The weight calculation formula is: W_m=α*||F_m||+β*F_i(k)+γ*P_m+δ*R_m. Among them, ||F_m|| is the L2 norm of the feature vector, calculated as √(Σ(F_m[j]^2)), j ranges from 1 to the dimension of the feature vector (here is 128). The L2 norm reflects the overall strength of the feature vector and can be regarded as a measure of feature significance. The larger the norm of the feature vector, the more significant the characteristics of the abnormal component signal are, which may represent a more important or obvious abnormal pattern.

[0219] F_i(k) is the screening factor, which reflects the overall abnormality of the signal.

[0220] P_m is the relative position importance of the signal in the trajectory, calculated as P_m=1-|2k_m / T-1|, where k_m is the index of the center frame of the abnormal component signal A_m, and T is the total number of frames. This formula gives higher position importance to the anomaly in the middle of the trajectory, while lower importance to the anomaly at the beginning and end.

[0221] R_m is the trajectory correlation factor, calculated as R_m=corr(A_m,T_m), where T_m is the part of the initial motion trajectory sequence corresponding to A_m, corr() represents the Pearson correlation coefficient, and the calculation process is as follows: First, calculate the average values ​​of A_m and T_m, denoted as μ_A and μ_T. Then, for each time point i, calculate the difference between A_m[i] and μ_A, and the difference between T_m[i] and μ_T, and multiply the two differences. Sum these products for all time points to get the covariance. Next, calculate the standard deviations of A_m and T_m, denoted as σ_A and σ_T, respectively. Finally, divide the covariance by the product of the standard deviations and the number of samples to get the correlation coefficient. The range of the calculated correlation coefficient R_m is [-1,1]. When R_m is close to 1, it means that A_m and T_m are highly positively correlated, that is, the change trend of the abnormal component signal is consistent with the original trajectory; when R_m is close to -1, it means that they are highly negatively correlated, that is, the change trend is opposite; when R_m is close to 0, it means that there is no obvious linear relationship between the two.

[0222] α, β, and γ are weight coefficients used to balance the importance of different factors.

[0223] This weight calculation method ensures that in the subsequent trajectory correction, attention can be reasonably allocated, focusing on those abnormal component signals that have obvious characteristics, high abnormality, are located in the key stages of movement, and are highly correlated with the initial trajectory.

[0224] Step S180, determining the correction weight of the abnormal component signal according to the screening factor and the weight, and correcting the initial motion trajectory sequence based on the correction weight to obtain a corrected motion trajectory sequence.

[0225] First, for each abnormal component signal A_m, the calculation formula of its correction weight MW_m is:

[0226] MW_m=W_m*(1+λ*F_i(k))

[0227] In this formula, W_m is the local weight obtained in the above steps, which reflects the importance of the abnormal component signal in the overall movement. F_i(k) is the screening factor obtained in the above steps, which indicates the degree of abnormality of the signal. λ is an adjustment parameter used to control the influence of the screening factor on the final correction weight. The choice of λ needs to be adjusted according to the specific application scenario. A larger λ value will make the influence of the screening factor more significant, while a smaller λ value will weaken this influence.

[0228] After calculating the correction weights of all abnormal component signals, normalization is required. The normalization formula is as follows:

[0229] NMW_m=MW_m / Σ(MW_j)(j from 1 to M)

[0230] Where M is the total number of abnormal component signals. This step ensures that the sum of all weights is 1, which helps to maintain the overall balance in the subsequent correction process and prevent some abnormal component signals from excessively affecting the final result.

[0231] Next, the calculated normalized correction weights are used to correct the initial motion trajectory. This correction process is performed frame by frame and key point by key point. For each key point in the trajectory, the influence of all abnormal component signals needs to be considered, and the position of the key point is adjusted according to their correction weights.

[0232] For the i-th key point in the k-th frame, the calculation formula for its corrected position p_ki_corrected is:

[0233] p_ki_corrected=p_ki+Σ(NMW_m*Δp_m_ki)(m from 1 to M)

[0234] In this formula, p_ki is the coordinate of the i-th human key point in the k-th frame in the initial motion trajectory sequence in the above step, and Δp_m_ki is the correction amount recommended by the m-th abnormal component signal for the human key point. In this way, each abnormal component signal affects the final correction result according to its importance and abnormality.

[0235] The calculation of the correction Δp_m_ki is based on the difference between the abnormal component signal and the original trajectory:

[0236] Δp_m_ki=p_m_ki-p_ki

[0237] Where p_m_ki is the corresponding coordinate in the abnormal component signal A_m. This correction reflects how the abnormal component signal thinks the key point should move to get closer to the "ideal" motion mode.

[0238] After applying the correction, a series of processing steps are required to ensure that the final trajectory conforms to the ideal motion pattern and satisfies the basic characteristics of motion. The first is the velocity constraint, which ensures that the corrected motion speed does not exceed a reasonable range. For each key point, calculate its velocity:

[0239] v_ki=(p_ki_corrected-p_(k-1)i_corrected) / Δt

[0240] Where Δt is the time interval between adjacent frames. If the calculated speed exceeds the preset maximum allowed speed v_max, an adjustment is required:

[0241] if|v_ki|>v_max:

[0242] v_ki=v_max*v_ki / |v_ki|

[0243] p_ki_constrained=p_(k-1)i_corrected+v_ki*Δt

[0244] else:

[0245] p_ki_constrained=p_ki_corrected

[0246] This adjustment ensures that the corrected movement speed is always within a reasonable range, avoiding fast movements that do not conform to the laws of physics.

[0247] Next, consider the acceleration constraint to ensure that the acceleration does not exceed the limit that the human body can reach. Calculate the acceleration:

[0248] a_ki=(v_ki-v_(k-1)i) / Δt

[0249] If the acceleration exceeds the preset maximum permissible acceleration a_max, adjustments are made:

[0250] if|a_ki|>a_max:

[0251] a_ki=a_max*a_ki / |a_ki|

[0252] v_ki_new=v_(k-1)i+a_ki*Δt

[0253] p_ki_constrained=p_(k-1)i_constrained+v_ki_new*Δt

[0254] This step further ensures the smoothness and continuity of the movement, avoiding sudden and drastic acceleration or deceleration.

[0255] In addition to physical constraints, physiological constraints also need to be considered to ensure that the joint angles do not exceed the normal range of motion. For adjacent key points (such as elbows and wrists), calculate the angle between them:

[0256] θ_ki=arctan2(p_ki_constrained.y-p_kj_constrained.y,

[0257] p_ki_constrained.x-p_kj_constrained.x)

[0258] Where j is the key point adjacent to i. If the calculated angle exceeds the preset range [θ_min,θ_max], make adjustments:

[0259] ifθ_ki<θ_minorθ_ki>θ_max:

[0260] θ_ki_constrained=max(θ_min,min(θ_max,θ_ki))

[0261] r=||p_ki_constrained-p_kj_constrained||

[0262] p_ki_constrained.x=p_kj_constrained.x+r*cos(θ_ki_constrained)

[0263] p_ki_constrained.y=p_kj_constrained.y+r*sin(θ_ki_constrained)

[0264] This adjustment ensures that the corrected movement posture still conforms to the human body's physiological structure and avoids unnatural or impossible joint angles.

[0265] Through this series of corrections and constraint processing, the corrected motion trajectory sequence is finally obtained:

[0266] Γ_corrected={p_ki_constrained|k=1,...,T;i=1,...,N}

[0267] Where T is the total number of frames and N is the number of key points. This new trajectory sequence should be closer to the "ideal" motion pattern while still retaining the key features of the original motion. It not only reflects the potential for improvement revealed by the abnormal component signal, but also conforms to the physical laws and basic characteristics of human motion.

[0268] Step S190, compare the corrected motion trajectory sequence with the preset standard action template, calculate the deviation values ​​of the student's body key points and limbs, and generate a comprehensive evaluation report based on the deviation values ​​and preset scoring rules.

[0269] First, the corrected motion trajectory sequence Γ_corrected is aligned with the preset standard motion template Γ_standard. The alignment process uses the Dynamic Time Warping (DTW) algorithm, which can handle the situation where the two time series have different lengths or inconsistent speeds and find the best matching method.

[0270] After alignment is completed, the deviation values ​​of each human key point and limb segment are calculated. For each key point i and each time step k, its position deviation is calculated:

[0271] δ_position_ki=||p_ki_constrained-p_ki_standard||

[0272] Where p_ki_constrained is the position in the corrected trajectory Γ_corrected, and p_ki_standard is the corresponding position in the standard template Γ_standard. This deviation value reflects the degree of difference between the motion trajectory and the standard template in spatial position.

[0273] At the same time, the angle deviation is calculated. For each joint (defined by two adjacent human key points), the angle deviation is calculated:

[0274] δ_angle_kij=|θ_kij_constrained-θ_kij_standard|

[0275] Where θ_kij_constrained is the joint angle in the corrected trajectory, and θ_kij_standard is the corresponding angle in the standard template. The angle deviation reflects the accuracy of the action posture.

[0276] For limb segments (such as upper arm, forearm, etc.), calculate the length deviation:

[0277] δ_length_kij=||l_kij_constrained-l_kij_standard|| / l_kij_standard

[0278] Where l_kij_constrained is the length of the limb segment in the corrected trajectory, and l_kij_standard is the corresponding length in the standard template. The length deviation reflects the change in body proportions during the action.

[0279] Based on these deviation values ​​and preset scoring rules, a comprehensive evaluation report is generated. The scoring rules include:

[0280] 1. Position accuracy score: S_position=Σ(w_i*exp(-δ_position_ki / σ_position))

[0281] 2. Angle accuracy score: S_angle=Σ(w_ij*exp(-δ_angle_kij / σ_angle))

[0282] 3. Limb proportion score: S_length=Σ(w_ij*exp(-δ_length_kij / σ_length))

[0283] Where w_i and w_ij are weight coefficients and σ is the standard deviation parameter used to control the sensitivity of the scores. These scores reflect the accuracy of the actions in different aspects.

[0284] The overall rating is a weighted average of these sub-scores:

[0285] S_total=(α*S_position+β*S_angle+γ*S_length) / (α+β+γ)

[0286] Among them, α, β, and γ are weight coefficients, reflecting the importance of different aspects in the overall evaluation.

[0287] The key action comparison part identifies the key frames or key stages in the standard template, and then finds the corresponding parts in the corrected trajectory for detailed comparison. Taking the throwing action as an example, the focus is on comparing the posture and speed in the preparation phase, acceleration phase, and release phase.

[0288] Specific indicators are calculated for each key stage:

[0289] 1. Body tilt angle in the preparation phase:

[0290] θ_tilt=arccos((p_shoulder_constrained-p_hip_constrained)·(0,1))

[0291] 2. Elbow joint angular velocity during acceleration:

[0292] ω_elbow=(θ_elbow_constrained_k-θ_elbow_constrained_(k-1)) / Δt

[0293] 3. Wrist position and speed during release phase:

[0294] v_wrist=(p_wrist_constrained_k-p_wrist_constrained_(k-1)) / Δt

[0295] Among them, p_shoulder_constrained, p_hip_constrained, and p_wrist_constrained are the position coordinates of the shoulder, hip, and wrist in the correction trajectory respectively.

[0296] Based on these calculation results, improvement suggestions are generated. For example, if the body tilt angle in the preparation phase deviates greatly, it is recommended to adjust the preparation posture and keep the body more upright to obtain better power transmission. If the elbow angular velocity in the acceleration phase is lower than the standard, it is recommended to increase the explosive power of the arm swing. If there is a significant deviation in the wrist position or speed in the release phase, it is recommended to slightly raise the arm at the moment of release to obtain a better parabolic trajectory.

[0297] The structure of the comprehensive evaluation report includes overall score, sub-score, key action comparison, detailed deviation data and improvement suggestions. The overall score gives S_total and its explanation, and the sub-score lists S_position, S_angle, S_length and their explanations. The key action comparison section includes detailed evaluation of the preparation phase, acceleration phase and release phase. The detailed deviation data shows the deviation of each joint and limb segment over time in the form of a chart. Improvement suggestions give specific improvement directions and training suggestions based on the above analysis.

[0298] The present application comprises the following steps: collecting initial video data for student training, segmenting the initial video data to obtain multiple initial video frames, and calculating the displacement of feature points between adjacent initial video frames to obtain multiple feature point displacement data; extracting background motion features of the initial video frames based on the feature point displacement data to obtain a background motion feature set; performing time series correlation analysis on the feature point displacement data and the background motion feature set to obtain a global motion estimation matrix; performing motion compensation on the initial video frames based on the global motion estimation matrix to obtain a stable video data sequence; performing human body key point detection and tracking on the stable video data sequence to obtain an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain motion component signals; and utilizing the motion component signals to obtain a stable video data sequence. number, analyze the position difference and time interval of adjacent human key points, determine the screening factor of the motion component signal, and screen the motion component signal according to the screening factor to obtain the abnormal component signal; extract the features of the abnormal component signal to obtain the weight feature vector; determine the weight corresponding to the abnormal component signal according to the weight feature vector, the screening factor and the initial motion trajectory sequence; determine the correction weight of the abnormal component signal according to the screening factor and the weight, and correct the initial motion trajectory sequence based on the correction weight to obtain the corrected motion trajectory sequence; compare the corrected motion trajectory sequence with the preset standard action template, calculate the deviation value of the student's human key points and limbs, and generate a comprehensive evaluation report based on the deviation value and the preset scoring rules. This application can obtain accurate feedback on the action in a timely manner by analyzing the students' actions in the initial video data, and efficiently identify the individual differences and specific needs of each student, so as to formulate a personalized training plan, thereby improving the training quality.

[0299] (1) This application introduces a global motion estimation matrix and motion compensation technology, which significantly improves the applicability and accuracy of school sports skills assessment. The global motion estimation matrix uses a Kalman filter to perform time-series correlation analysis, comprehensively considers background motion characteristics and feature point displacement data, and can accurately capture the motion information of the entire scene. This technology performs well in outdoor environments such as sprint training on a track, and can accurately identify and separate background motion even in the presence of interference factors. For events with large movements such as long jump, this technology can effectively handle camera shake caused by take-off and landing, ensuring that the accurate motion trajectory is captured.

[0300] Motion compensation technology creates a virtual stable perspective by applying an inverse transformation to each frame of the image, allowing coaches to clearly observe the details of the movement without being affected by the slight shake of the camera. For projects such as pull-ups that require multi-angle observation, the system can reconstruct a stable movement sequence and accurately analyze the changes in body posture throughout the process. The application of these technologies expands the use scenarios of the system in various school environments, and high-quality movement data can be obtained without the need for professional equipment.

[0301] (2) In the signal processing stage, this application uses a multi-dimensional signal decomposition and feature extraction method to provide in-depth motion analysis capabilities for school physical education. Frequency domain analysis through Fourier transform can effectively evaluate the rhythm of repetitive movements such as running and skipping. Time-frequency analysis captures the changes in motion characteristics over time, which is particularly useful in analyzing the take-off moment of long jump or the pull-up stage, and can accurately locate the time point of power burst.

[0302] Empirical mode decomposition decomposes complex nonlinear motion signals into intrinsic mode functions, which helps to separate motion components of different frequencies, such as arm swing, rope rotation and jumping in rope skipping, and helps identify coordination problems. Principal component analysis simplifies the representation of high-dimensional motion data and highlights the most recognizable motion features, such as the change in the angle between the upper arm and the torso or the stability of the shoulder in pull-ups. This comprehensive analysis method can accurately identify abnormal movements and key technical details, providing coaches with a comprehensive and in-depth technical analysis foundation.

[0303] (3) This application designs a method for calculating the weight of abnormal component signals based on multiple factors and applies it to the correction of motion trajectories, thereby improving the accuracy and personalization of motion evaluation. The system considers multiple factors such as the positional importance of the abnormal motion in the overall technology, kinematic characteristics, and the degree of deviation from the standard motion to calculate the weight.

[0304] For example, when evaluating long jump, abnormalities at the moment of take-off will be given a higher weight. The system also takes into account individual characteristics of students, such as height and weight, to avoid simply considering all movements that deviate from the standard pattern as errors. For running, the system assigns weights according to the importance of different stages, while taking into account the physical condition and endurance level of the students. In projects such as skipping and pull-ups, the system pays special attention to key movement stages, such as the coordination of arms and legs, and the explosive power in the early stage of pull-up.

[0305] This meticulous weighting mechanism ensures that the evaluation results accurately reflect each student's unique performance. During the trajectory correction phase, the system applies these weights to optimize the movement trajectory and ensure the standardization of technical movements. Ultimately, this intelligent movement optimization lays the foundation for generating accurate and targeted skill evaluation reports.

[0306] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0307] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 2 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as initial video and initial video frame. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a computer vision sports skill evaluation method based on signal processing is implemented.

[0308] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0309] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0310] Collecting initial video data for student training, segmenting the initial video data to obtain multiple initial video frames, and calculating the displacement of feature points between adjacent initial video frames to obtain multiple feature point displacement data;

[0311] Extract the background motion features of the initial video frame according to the feature point displacement data to obtain a background motion feature set;

[0312] Perform temporal correlation analysis on feature point displacement data and background motion feature set to obtain a global motion estimation matrix;

[0313] Based on the global motion estimation matrix, motion compensation is performed on the initial video frame to obtain a stable video data sequence;

[0314] Performing human key point detection and tracking on the stable video data sequence to obtain an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain motion component signals;

[0315] Using the motion component signal, analyzing the position difference and time interval of adjacent key points of the human body, determining the screening factor of the motion component signal, and screening the motion component signal according to the screening factor to obtain the abnormal component signal;

[0316] Extract features of abnormal component signals to obtain weighted feature vectors; determine local weights corresponding to abnormal component signals based on the weighted feature vectors, screening factors, and initial motion trajectory sequences;

[0317] According to the screening factors and weights, the correction weights of the abnormal component signals are determined, and based on the correction weights, the initial motion trajectory sequence is corrected to obtain a corrected motion trajectory sequence;

[0318] The corrected motion trajectory sequence is compared with the preset standard action template, the deviation values ​​of the students' key points and limbs are calculated, and a comprehensive evaluation report is generated based on the deviation values ​​and preset scoring rules.

[0319] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0320] Collecting initial video data for student training, segmenting the initial video data to obtain multiple initial video frames, and calculating the displacement of feature points between adjacent initial video frames to obtain multiple feature point displacement data;

[0321] Extract the background motion features of the initial video frame according to the feature point displacement data to obtain a background motion feature set;

[0322] Perform temporal correlation analysis on feature point displacement data and background motion feature set to obtain a global motion estimation matrix;

[0323] Based on the global motion estimation matrix, motion compensation is performed on the initial video frame to obtain a stable video data sequence;

[0324] Performing human key point detection and tracking on the stable video data sequence to obtain an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain motion component signals;

[0325] Using the motion component signal, analyzing the position difference and time interval of adjacent key points of the human body, determining the screening factor of the motion component signal, and screening the motion component signal according to the screening factor to obtain the abnormal component signal;

[0326] Extract features of abnormal component signals to obtain weighted feature vectors; determine local weights corresponding to abnormal component signals based on the weighted feature vectors, screening factors, and initial motion trajectory sequences;

[0327] According to the screening factors and weights, the correction weights of the abnormal component signals are determined, and based on the correction weights, the initial motion trajectory sequence is corrected to obtain a corrected motion trajectory sequence;

[0328] The corrected motion trajectory sequence is compared with the preset standard action template, the deviation values ​​of the students' key points and limbs are calculated, and a comprehensive evaluation report is generated based on the deviation values ​​and preset scoring rules.

[0329] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0330] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0331] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A computer vision sports skills assessment method based on signal processing, characterized in that: The following steps are involved: Collecting initial video data of student training, segmenting the initial video data to obtain multiple initial video frames, and calculating the displacement of feature points between adjacent initial video frames to obtain multiple feature point displacement data; Extracting background motion features of the initial video frame according to the feature point displacement data to obtain a background motion feature set; Performing temporal correlation analysis on the feature point displacement data and the background motion feature set to obtain a global motion estimation matrix; Based on the global motion estimation matrix, performing motion compensation on the initial video frame to obtain a stable video data sequence; Performing human body key point detection and tracking on the stable video data sequence to obtain an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain motion component signals; Utilizing the motion component signal, analyzing the position difference and time interval of adjacent key points of the human body, determining a screening factor of the motion component signal, and screening the motion component signal according to the screening factor to obtain an abnormal component signal; Extracting features of the abnormal component signal to obtain a weighted feature vector; Determining a local weight corresponding to the abnormal component signal according to the weight feature vector, the screening factor and the initial motion trajectory sequence; Determining a correction weight of the abnormal component signal according to the screening factor and the weight, and correcting the initial motion trajectory sequence based on the correction weight to obtain a corrected motion trajectory sequence; Compare the modified motion trajectory sequence with the preset standard action template, calculate the deviation values ​​of the key points and limbs of the student's body, and generate a comprehensive evaluation report based on the deviation values ​​and preset scoring rules; The steps of detecting and tracking human key points on the stable video data sequence to obtain an initial motion trajectory sequence, and decomposing the initial motion trajectory sequence to obtain motion component signals include the following steps: Using a deep learning model, a group of human key points are respectively identified in each of the stable video data sequences, and the detection coordinates of the human key points are obtained; the human key points identified in each of the stable video data sequences are the same; Predicting the predicted coordinates of the key points of the human body in the stable video data sequence of the current frame in the stable video data sequence of the next frame based on the Kalman filter; Using the Hungarian algorithm, the predicted coordinates are matched with the detected coordinates of the key points of the human body in the stable video data sequence of the next frame to obtain an initial motion trajectory sequence of the key points of the human body in the stable video data sequence; Decomposing the initial motion trajectory sequence to obtain motion component signals; The step of decomposing the initial motion trajectory sequence to obtain motion component signals includes the following steps: Convert the detected coordinates of the key points of the human body from the image coordinate system to the body coordinate system with the center of the hip as the origin to obtain the converted coordinates; Convert the transformed coordinates from the time series to the frequency domain using Fourier transform to obtain a frequency domain representation of the transformed coordinates; Transforming the transformation coordinates using wavelet transform to obtain a time-frequency representation of the transformation coordinates; Decomposing the transformed coordinates using empirical mode decomposition to obtain an intrinsic mode function of the transformed coordinates; Performing principal component analysis on the initial motion trajectory sequence to obtain principal components; Combining the frequency domain representation, the time-frequency representation, the intrinsic mode function and the principal component into the motion component signal; The step of using the motion component signal to analyze the position difference and time interval of adjacent human body key points, determining the screening factor of the motion component signal, and screening the motion component signal according to the screening factor to obtain the abnormal component signal includes the following steps: Calculating the Euclidean distances between adjacent human key points in each of the initial video frames, and calculating the mean and standard deviation of the Euclidean distances; Calculating the displacement of the key points of the human body in the continuous initial video frames, and calculating the mean and standard deviation of the displacement; According to the motion component signal, identifying the peak frequency of the main frequency component of the key point of the human body in each of the initial video frames, and calculating the mean and standard deviation of the peak frequency; According to the motion component signal, calculating the time-frequency energy of the key points of the human body in each of the initial video frames, and calculating the mean and standard deviation of the time-frequency energy; Calculating the instantaneous amplitude of the key point of the human body in each of the initial video frames according to the motion component signal, and calculating the mean and standard deviation of the instantaneous amplitude; According to the motion component signal, the initial motion trajectory sequence is projected into the principal component space to obtain projection coefficients, and the mean and standard deviation of the projection coefficients are calculated; Constructing the screening factor according to the Euclidean distance, the mean and standard deviation of the Euclidean distance, the displacement, the mean and standard deviation of the displacement, the peak frequency, the mean and standard deviation of the peak frequency, the time-frequency energy, the mean and standard deviation of the time-frequency energy, the instantaneous amplitude, the mean and standard deviation of the instantaneous amplitude, the projection coefficient and the mean and standard deviation of the projection coefficient; When the screening factor corresponding to the motion component signal is greater than a preset threshold, the motion component signal with the screening factor greater than the preset threshold is the abnormal component signal.

2. The computer vision sports skills assessment method based on signal processing according to claim 1, characterized in that: The step of extracting the background motion features of the initial video frame according to the feature point displacement data to obtain a background motion feature set comprises the steps of: Analyzing the feature point displacement data based on a random sampling consistency algorithm to obtain a background motion feature model between each pair of adjacent initial video frames; Detecting a set of background feature points from the initial video frame; Based on the background motion feature model, predict the predicted position of the background feature point of the previous initial video frame between the corresponding adjacent initial video frames in the next initial video frame, and calculate the error between the predicted position and the actual position of the background feature point of the next initial video frame; According to the error, a background motion feature vector corresponding to the initial video frame is defined, and the background motion feature set is constructed based on the background motion feature vector.

3. The computer vision sports skills assessment method based on signal processing according to claim 1, characterized in that: The step of performing temporal correlation analysis on the feature point displacement data and the background motion feature set to obtain a global motion estimation matrix comprises the following steps: Extracting a displacement vector from the feature point displacement data, and extracting a statistic from the background motion feature set; Calculating initial affine transformation parameters according to the displacement vector, adjusting the initial affine transformation parameters based on the statistics, acquiring affine transformation parameters, and constructing an observation vector according to the affine transformation parameters; Based on the observation vector and the Kalman filter, the global motion estimation matrix is ​​output.

4. The computer vision sports skills assessment method based on signal processing according to claim 1, characterized in that: The step of performing motion compensation on the initial video frame based on the global motion estimation matrix to obtain a stable video data sequence comprises the steps of: Selecting the first frame in the initial video data as a reference frame; Based on the global motion estimation matrix, calculating the cumulative transformation matrix from the reference frame to any of the initial video frames; The initial video frame is stabilized based on the inverse matrix of the cumulative transformation matrix to obtain the stabilized video data sequence.

5. The computer vision sports skills assessment method based on signal processing according to claim 1, characterized in that: The step of extracting features from the abnormal component signal to obtain a weighted feature vector comprises the following steps: Preprocessing the abnormal component signal; Input the preprocessed abnormal component signal into a temporal convolutional network and output a feature map; The feature map is passed through a global average pooling layer to compress the time dimension and obtain a global feature vector; The global feature vector is processed through a fully connected layer to output a weighted feature vector.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images