Sports motion error correction method, device and equipment based on computer vision and medium

Abstract

The invention relates to a sports motion error correction method and device based on computer vision, equipment and a medium. The method comprises the following steps: firstly, acquiring visual motion video data of a target user in a sports process, extracting a visual spatial-temporal feature vector through a preset visual feature extraction network, further generating a disturbance visual feature vector, calculating a joint feature drift vector and an error perception statistical magnitude through a micro-error disturbance model, and calculating the visual motion video data according to the joint feature drift vector and the error perception statistical magnitude. And fusing the visual spatio-temporal feature vector and the error perception statistic to generate a fused feature vector, and generating an error correction instruction according to the fused feature vector. By adopting the method, through active evaluation and integration of stability information of visual features, negative conduction of underlying perception uncertainty to a final decision instruction is effectively inhibited, and the reliability of error correction guidance in a complex real scene is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method, device, equipment and medium for correcting sports movements based on computer vision. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, vision-based motion capture and analysis has become an important tool in sports science training. Acquiring athlete videos via cameras and using deep learning models to automatically analyze movements and postures, identify technical errors, and provide real-time feedback and correction guidance is currently the mainstream research direction in the field of sports motion correction.

[0003] Traditional computer vision-based sports motion correction methods typically follow a direct "perception-decision" paradigm: first, pose estimation algorithms are used to extract the spatiotemporal coordinates or high-level feature vectors of human joints from video sequences; then, these features are directly input into a classifier or generative model, mapping them to predefined error categories or specific error correction instructions. This method relies entirely on the accuracy of the visual information extracted from the video.

[0004] However, in real-world complex motion scenarios, the visual perception process inherently contains unavoidable uncertainties. For example, factors such as motion blur caused by high-speed motion, limb occlusion, variable lighting conditions, and clothing interference can all lead to deviations or jitter in pose estimation results. Existing methods treat these potentially erroneous visual features as deterministic and reliable input, directly using them for final decision-making, lacking an assessment and response mechanism for "perceptual uncertainty." This can lead to the system potentially generating a correction instruction based on a slightly flawed understanding of the action, even if it is incorrect in key details or unrelated to the actual error. For example, it might diagnose "insufficient elbow angle" due to occlusion as "incorrect force application timing." This negative transmission from low-level perceptual errors to high-level semantic instructions severely affects the reliability and practicality of error correction, representing a core technical problem that current pure vision methods urgently need to address. Summary of the Invention

[0005] Therefore, it is necessary to provide computer vision-based methods, devices, equipment, and media for correcting sports movements in order to address the aforementioned technical problems.

[0006] Firstly, this application provides a computer vision-based method for correcting errors in sports movements, including:

[0007] S1. Acquire visual motion video data of the target user during sports activities, and input the visual motion video data into a preset visual feature extraction network to output a visual spatiotemporal feature vector; the visual feature extraction network includes a three-dimensional convolutional neural network, a spatiotemporal attention layer, and a fully connected layer;

[0008] S2. Based on the visual spatiotemporal feature vector, a perturbation visual feature vector is generated through preset perturbation parameter configuration;

[0009] S3. Input the difference between the visual spatiotemporal feature vector and the perturbation visual feature vector into the preset differentiable error perturbation model for processing to obtain multiple predicted joint feature drift vectors.

[0010] S4. Based on the predicted joint feature drift vectors, calculate the error perception statistic; whereby the error perception statistic is used to characterize the stability of the visual spatiotemporal feature vector under perturbation.

[0011] S5. The visual spatiotemporal feature vector and the error perception statistics are fused to generate a fused feature vector, and an error correction instruction is generated based on the fused feature vector; wherein, the error correction instruction is used to instruct the user to adjust the sports action.

[0012] In one embodiment, the pre-defined differentiable error perturbation model is obtained by the following method:

[0013] S11. Obtain a multimodal training dataset consisting of visual video data, inertial measurement unit data, and electromyographic signal data that are strictly synchronized with time stamps; each sample in the multimodal training dataset is associated with an expert-annotated action error category label;

[0014] S12. Using the multimodal training dataset, train a multimodal fusion encoder to obtain the trained multimodal fusion encoder. The multimodal fusion encoder takes visual feature vectors and non-visual feature vectors as inputs, performs feature fusion through an internal multi-head cross-attention mechanism, and outputs the fused multimodal joint feature vector.

[0015] S13. Select a training sample from the multimodal training dataset, extract the visual feature vector of the training sample through the visual feature extraction network, and calculate the standard joint feature vector of the training sample by combining the non-visual feature vector of the training sample with the trained multimodal fusion encoder.

[0016] S14. Apply various artificial perturbations to the visual feature vectors to generate multiple perturbed visual feature vectors. Input each perturbed visual feature vector and the non-visual feature vector of the same sample into the trained multimodal fusion encoder to obtain the corresponding biased joint feature vector.

[0017] S15. Calculate the first difference vector between the visual feature vector and each perturbed visual feature vector, and calculate the second difference vector between the standard joint feature vector and each biased joint feature vector.

[0018] S16. Construct a differentiable model of a multilayer perceptron as an error perturbation simulator, take the first difference vector as input and the second difference vector as supervision target, train the error perturbation simulator by minimizing the mean square error loss function between the model prediction output and the supervision target, so that the error perturbation simulator approximates the propagation function from visual feature perturbation to joint feature bias, and obtain the preset differentiable error perturbation model after training.

[0019] In one embodiment, visual motion video data is input into a preset visual feature extraction network, which outputs a visual spatiotemporal feature vector, including:

[0020] S21. Perform frame sampling and time alignment processing on the visual motion video data to obtain a standardized video frame sequence;

[0021] S22. Spatiotemporal feature extraction is performed on the video frame sequence using a three-dimensional convolutional neural network to obtain a preliminary spatiotemporal feature map;

[0022] S23. Perform pooling operation on the time dimension on the preliminary spatiotemporal feature map to obtain the temporal aggregated features;

[0023] S24. Weighted spatiotemporal features are obtained by weighting the temporal aggregation features through a spatiotemporal attention layer.

[0024] S25. The weighted spatiotemporal features are mapped through a fully connected layer to obtain the visual spatiotemporal feature vector.

[0025] In one embodiment, based on the visual spatiotemporal feature vector, a perturbation visual feature vector is generated through a preset perturbation parameter configuration, including:

[0026] S31. Standardize the visual spatiotemporal feature vector to obtain the standard feature vector;

[0027] S32. Generate multiple disturbance vectors according to the preset disturbance parameter configuration; the preset disturbance parameter configuration includes disturbance intensity parameters and disturbance direction matrix;

[0028] S33. Add each perturbation vector to the standard eigenvector to obtain multiple perturbation eigenvectors;

[0029] S34. Perform inverse normalization on each perturbation feature vector to obtain multiple perturbation visual feature vectors.

[0030] In one embodiment, the difference between the visual spatiotemporal feature vector and the perturbed visual feature vector is input into a preset differentiable error perturbation model for processing to obtain multiple predicted joint feature drift vectors, including:

[0031] S41. Calculate the difference feature between the visual spatiotemporal feature vector and each perturbation visual feature vector;

[0032] S42. Input each difference feature into the preset differentiable error perturbation model to obtain the predicted joint feature drift vector;

[0033] S43. Normalize each predicted joint feature drift vector to obtain a normalized joint feature drift vector.

[0034] S44. Based on the preset drift intensity coefficient, scale each normalized joint feature drift vector to obtain multiple predicted joint feature drift vectors.

[0035] In one embodiment, an error-aware statistic is calculated based on multiple predicted joint feature drift vectors, including:

[0036] S51. Calculate the covariance matrix of multiple joint feature drift vectors to obtain the drift covariance matrix;

[0037] S52. Perform eigenvalue decomposition on the drift covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors;

[0038] S53. Based on the eigenvalues, the error perception statistic is obtained; whereby the expression for the error perception statistic is:

[0039]

[0040] In the formula, This represents the error perception statistic. For the first 1 eigenvalue, Represents the total number of eigenvalues. It is a very small constant.

[0041] In one embodiment, a fused feature vector is generated by fusing visual spatiotemporal feature vectors and error perception statistics, and an error correction instruction is generated based on the fused feature vector, including:

[0042] S61. Extend the error perception statistics to an error perception vector with the same dimension as the visual spatiotemporal feature vector.

[0043] S62. The visual spatiotemporal feature vector and the error perception vector are fused by element-wise multiplication to obtain a weighted visual feature vector.

[0044] S63. Input the weighted visual feature vector into the preset action classifier to obtain the probability distribution of action error categories;

[0045] S64. Based on the probability distribution of action error categories, select the action error category with the highest probability;

[0046] S65. Based on the action error category, query the preset error correction instruction library and generate the corresponding error correction instruction; wherein, the error correction instruction includes an error description and adjustment suggestions.

[0047] Secondly, this application also provides a computer vision-based sports motion correction device, comprising:

[0048] The data acquisition and feature extraction module is used to acquire visual motion video data of the target user during sports activities, and input the visual motion video data into a preset visual feature extraction network to output a visual spatiotemporal feature vector; the visual feature extraction network includes a three-dimensional convolutional neural network, a spatiotemporal attention layer and a fully connected layer;

[0049] The perturbation generation module is used to generate perturbation visual feature vectors based on visual spatiotemporal feature vectors and through preset perturbation parameter configurations.

[0050] The perturbation error analysis module is used to input the difference features between the visual spatiotemporal feature vector and the perturbation visual feature vector into the preset differentiable error perturbation model for processing, and to obtain multiple predicted joint feature drift vectors.

[0051] The error calculation module is used to calculate the error perception statistics based on multiple joint feature drift vectors predicted; the error perception statistics are used to characterize the stability of visual spatiotemporal feature vectors under perturbation.

[0052] The error correction instruction generation module is used to fuse visual spatiotemporal feature vectors and error perception statistics to generate fused feature vectors, and to generate error correction instructions based on the fused feature vectors; the error correction instructions are used to instruct users to adjust their sports movements.

[0053] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0054] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0055] The aforementioned computer vision-based sports motion correction method, device, equipment, and medium first acquire visual motion video data of the target user during sports activities, input it into a visual feature extraction network composed of a 3D convolutional neural network, a spatiotemporal attention layer, and a fully connected layer to output a visual spatiotemporal feature vector, then generate a perturbed visual feature vector based on the visual spatiotemporal feature vector and preset perturbation parameters, next calculate the difference feature between the visual spatiotemporal feature vector and the perturbed visual feature vector and input it into a preset differentiable error perturbation model to obtain multiple predicted joint feature drift vectors, then calculate the error perception statistics to characterize the stability of the visual spatiotemporal feature vector perturbation based on these joint feature drift vectors, and finally fuse the visual spatiotemporal feature vector and the error perception statistics to generate a fused feature vector and generate correction instructions accordingly. This coherent approach effectively solves the problem of unassessed and unaddressed perceptual uncertainty in traditional pure vision sports motion correction methods, weakens the negative transmission of lower-level visual perception errors to higher-level correction instructions, improves the reliability and practicality of sports motion correction, and ensures that the generated correction instructions can accurately guide users to adjust their sports movements. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating a computer vision-based sports motion correction method in one embodiment.

[0058] Figure 2 This is a schematic diagram illustrating the construction process of a differentiable error perturbation model in one embodiment;

[0059] Figure 3 This is a schematic diagram of a computer vision-based sports motion correction device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] refer to Figure 1 The document presents a flowchart illustrating the computer vision-based sports motion correction method provided in this application, which includes the following steps:

[0062] S1. Acquire visual motion video data of the target user during sports activities, input the visual motion video data into a preset visual feature extraction network, and output a visual spatiotemporal feature vector.

[0063] The visual feature extraction network includes a 3D convolutional neural network, a spatiotemporal attention layer, and a fully connected layer.

[0064] Specifically, when acquiring visual motion video data of the target user during sports activities, an industrial-grade high-definition camera is used as the acquisition device. The acquisition angle is adaptively adjusted according to the target movement type to ensure complete capture of the movement trajectory of the human torso and limbs. During the acquisition process, an image acquisition card converts the analog video signal into a digital signal and stores it as a video file in a standard video format. After acquisition, the visual motion video data is preprocessed, including using Gaussian filtering to remove image noise. The Gaussian kernel function is set as follows: ,in, and These represent the coordinate offsets of image pixels in the horizontal and vertical directions, respectively. The standard deviation of the Gaussian kernel function is used to control the smoothness of the filtering. Video frames are uniformly scaled to a size suitable for subsequent network input using bilinear interpolation. Static background frames are removed using inter-frame differencing, retaining only valid frame sequences containing moving targets. The preprocessed video data is input into a pre-defined visual feature extraction network. This network's 3D convolutional neural network employs a typical 3D convolutional architecture, containing multiple convolutional and pooling layers. It extracts spatial and temporal motion features from the video frame sequence through 3D convolutional operations. The spatiotemporal attention layer uses a channel-space combined attention mechanism. First, the channel attention module adaptively adjusts the weights of each channel in the 3D convolution output feature map, using a combination of global average pooling and global max pooling for channel weight calculation. Then, the spatial attention module allocates spatial weights to the fused feature map, enhancing features in key regions such as human joints. The fully connected layer contains multiple hidden layers, employing the ReLU activation function, and ultimately outputs a visual spatiotemporal feature vector that accurately represents the spatiotemporal dynamics of human motion.

[0065] S2. Based on the visual spatiotemporal feature vector, a perturbation visual feature vector is generated through preset perturbation parameter configuration.

[0066] Specifically, when generating perturbed visual feature vectors based on spatiotemporal visual feature vectors, the preset perturbation parameter configuration is determined based on the visual interference characteristics of sports scenarios, specifically including a combination configuration of three perturbation types: Gaussian perturbation, uniform perturbation, and salt-and-pepper perturbation. The perturbation parameter configuration is stored in a standard configuration file, containing the perturbation type, the weight ratio of each type of perturbation, and specific parameter values. When generating the perturbed visual feature vector, firstly, based on the preset perturbation parameter configuration, a perturbation vector with the same dimensions as the spatiotemporal visual feature vector is generated using a numerical calculation library. The Gaussian perturbation vector is randomly generated using a normal distribution, the uniform perturbation vector is randomly generated using a uniform distribution, and the salt-and-pepper perturbation vector is generated by randomly selecting a portion of the feature dimensions and performing extreme value processing on their values. Then, the three perturbation vectors are weighted and fused according to preset weight ratios to obtain a comprehensive perturbation vector. Finally, the spatiotemporal visual feature vector and the comprehensive perturbation vector are added element-wise to generate the perturbed visual feature vector. This process is implemented through matrix addition operations to ensure that the perturbation effect conforms to the uncertainty characteristics of visual perception in actual scenarios.

[0067] S3. Input the difference between the visual spatiotemporal feature vector and the perturbation visual feature vector into the preset differentiable error perturbation model for processing to obtain multiple predicted joint feature drift vectors.

[0068] Specifically, firstly, the difference feature between the visual spatiotemporal feature vector and the perturbed visual feature vector is calculated. The difference feature is calculated using the element-wise difference method of L2 distance, that is, for the visual spatiotemporal feature vector... and perturbation visual feature vector Difference characteristics: , where each element , This is the dimension index of the feature vector. After calculation, the difference features are normalized using the Min-Max normalization method to map the difference features to a fixed interval. The normalization formula is: ,in, Let i be the original value of the i-th dimension of the difference feature. It is the minimum value among all dimensions of the difference feature. The normalization process, which sets the maximum value across all dimensions of the difference features, eliminates the impact of variations in feature value ranges across different dimensions on the model's performance. The pre-defined differentiable error perturbation model employs a generative model architecture based on a multilayer perceptron, containing multiple hidden layers. The activation function is Leaky ReLU, and the output layer uses a linear activation function. This model learns the mapping relationship between the difference features and the joint feature drift vector through a differentiable backpropagation mechanism. During training, mean squared error is used as the loss function, and an adaptive momentum optimizer is employed. After inputting the normalized difference features into the model, the model performs a non-linear mapping of the difference features through matrix multiplication operations of the multilayer perceptron. This mapping then passes through feature transformations and activation processes in each hidden layer, ultimately outputting multiple joint feature drift vectors, each corresponding to a potential type of visual perception error.

[0069] S4. Calculate the error-aware statistics based on the predicted drift vectors of multiple joint features.

[0070] Optionally, error-aware statistics are used to characterize the stability of visual spatiotemporal feature vectors under perturbation.

[0071] Specifically, when calculating the error perception statistic based on multiple predicted joint feature drift vectors, it is first clarified that the error perception statistic includes five core indicators: variance, mean, maximum drift value, minimum drift value, and drift vector correlation coefficient. These indicators collectively characterize the stability of the visual spatiotemporal feature vector under perturbation. Variance reflects the dispersion of the drift vector, the mean reflects the overall shift trend of the drift vector, the maximum / minimum drift value reflects the drift amplitude under extreme conditions, and the correlation coefficient reflects the drift correlation caused by different types of errors. The specific calculation process is as follows: For multiple joint feature drift vectors... , To determine the number of joint feature drift vectors, calculate the variance in each dimension. The variance formula is: ,in, For the first The variance of the dimensional features For the first The joint feature drift vector of the first The numerical value of the dimension. For the first The mean of each dimension, This represents the total number of joint feature drift vectors. The formula for calculating the mean is: The meanings of the parameters are consistent with those in the variance formula. Maximum drift value. For all joint feature drift vectors, the first... Maximum value and minimum drift value in dimensional values For all joint feature drift vectors, the first... The minimum value in the dimension. The correlation coefficient uses the Pearson correlation coefficient, calculated for any two drift vectors. and The correlation is expressed by the formula: ,in, For vectors and Pearson correlation coefficient, For vectors and covariance, For vectors standard deviation For vectors The standard deviation of the standard deviation is calculated. After calculation, the five indicators are concatenated according to their dimensions to form an error perception statistics matrix. Then, it is flattened into a one-dimensional vector, which is used as the input for subsequent feature fusion. This statistic can accurately quantify the stability of visual spatiotemporal features under different perturbations. If the variance is large and the maximum drift value deviates from the threshold, it indicates that the visual spatiotemporal feature is greatly affected by perceptual errors and has poor stability.

[0072] S5. The visual spatiotemporal feature vector and the error perception statistics are fused to generate a fused feature vector, and error correction instructions are generated based on the fused feature vector.

[0073] Optionally, the error correction instructions are used to instruct the user to adjust the sports movements.

[0074] Optionally, feature fusion employs an attention-weighted fusion mechanism, first processing the visual spatiotemporal feature vectors... and error-aware statistic vector Dimensional unification is performed by mapping the visual spatiotemporal feature vector to the same dimension as the error-aware statistical vector using a 1×1 convolutional layer. The number of convolutional kernels matches the target dimension, the stride is set to 1, and equal-length padding is used to obtain the mapped visual spatiotemporal feature vector. Then, an attention weight calculation module is introduced, which uses a single-hidden-layer perceptron, with the input being... and The element-wise product vector is used, the hidden layer has an appropriate number of neurons, the activation function is sigmoid, and the output is an attention weight vector with the same dimension as the fused feature. Each element of the weight vector The value ranges from 0 to 1, and is used to represent the first... The importance of each dimension of features. The formula for calculating the fused feature vector is: ,in, For the fusion feature vector of the th The numerical value of the dimension. For the first Attention weights of dimensions The mapped visual spatiotemporal feature vector is the first... The numerical value of the dimension. For the error-aware statistic vector, the first... The numerical value of the dimension. This weighted fusion method can highlight effective visual features and weaken features that are greatly affected by errors, thus improving the reliability of the fused features. The fused feature vector is input into a preset error correction instruction generation model. This model adopts a Transformer architecture and contains multiple encoder layers. Each encoder layer consists of a multi-head self-attention mechanism and a feedforward neural network. The corresponding word embedding dimension is set, and the model is trained using the cross-entropy loss function. The model output is a fixed-format error correction instruction text. The specific generation process is as follows: after the fused feature vector is processed by the encoder layer to extract high-level semantic features, the semantic features are mapped to a preset error correction instruction dictionary by the decoder. The dictionary contains key semantic units such as action parts, error types, and adjustment directions. Finally, the structured error correction instructions are generated by concatenating these instructions, which can accurately instruct users to make targeted adjustments to sports movements.

[0075] In the aforementioned computer vision-based sports motion correction method, the visual spatiotemporal feature vector representing the motion is first extracted from the visual motion video. Then, based on this feature vector, a set of perturbed feature versions is actively simulated and generated. The difference between the original features and the perturbed features is then input into a pre-trained differentiable error perturbation model to predict various drift scenarios that may occur in the semantic understanding (i.e., joint features) of the motion under simulated perturbation. Next, these predicted drifts are statistically calculated to obtain an error perception statistic that quantifies the stability of the original visual features. Finally, in the key step of generating correction instructions, instead of relying solely on the initial visual features, this stability statistic is fused with the original features, thereby guiding the model to generate a correction instruction that is inherently adaptable to the uncertainty of underlying visual perception. This effectively reduces the risk of misjudgment of instructions caused by inherent noise and errors in the video analysis process, and improves the reliability and practicality of motion correction suggestions.

[0076] In an optional embodiment, such as Figure 2 This is a schematic diagram illustrating the construction process of a differentiable error perturbation model in one embodiment, including the following steps:

[0077] S11. Obtain a multimodal training dataset consisting of visual video data, inertial measurement unit data, and electromyographic signal data that are strictly time-stamped.

[0078] Optionally, each sample in the multimodal training dataset is associated with an expert-annotated action error category label.

[0079] Optionally, when acquiring the multimodal training dataset, the dataset consists of visual video data, inertial measurement unit (IMU) data, and electromyography (EMG) signal data, with the three types of data strictly synchronized via timestamps. Visual video data is acquired using the same acquisition equipment and preprocessing procedures as described above, ensuring the data format is compatible with the subsequent feature extraction network. IMU data is acquired using IMU sensors worn on key movement areas of the target user. Acquisition parameters include kinematic parameters such as acceleration, angular velocity, and attitude angle. The sensor sampling frequency matches the frame rate of the visual video data to ensure time synchronization accuracy. EMG signal data is acquired using surface EMG sensors, which are attached to the surface of muscle groups related to the target movement, acquiring changes in electrical signals during muscle contraction. Baseline correction is performed on the EMG signals during acquisition to remove environmental electromagnetic interference. Each sample is associated with an expert-annotated movement error category label. The labeling process is completed by experts with extensive sports training experience. The label content includes key information such as the location of the erroneous movement, the error type, and the error level, forming a structured labeling system to provide supervision for subsequent model training. After the dataset is constructed, data cleaning is performed to remove samples with missing data, distorted signals, or ambiguous labels. At the same time, data augmentation techniques are used to expand the size of the dataset and improve the model's generalization ability.

[0080] S12. Using the multimodal training dataset, train a multimodal fusion encoder to obtain the trained multimodal fusion encoder.

[0081] Optionally, the multimodal fusion encoder takes visual feature vectors and non-visual feature vectors as input, performs feature fusion through an internal multi-head cross-attention mechanism, and outputs a fused multimodal joint feature vector.

[0082] Optionally, when training the multimodal fusion encoder using a multimodal training dataset, the encoder's input consists of visual and non-visual feature vectors. The non-visual feature vectors are obtained from inertial measurement unit (IMU) data and electromyography (EMG) signal data through feature extraction. Feature extraction of the IMU data is achieved using a sliding window method, dividing the acquired time-series data into fixed-length windows and calculating the statistical features within each window. Feature extraction of the EMG signal data employs wavelet transform, converting the time-domain signal to the frequency domain and extracting features at different frequency components. The features from the two types of non-visual data are then concatenated to form a non-visual feature vector. Internally, the multimodal fusion encoder uses a multi-head cross-attention mechanism for feature fusion. This mechanism calculates the association weights between the visual and non-visual feature vectors in parallel using multiple attention heads. The calculation of attention weights is based on a query-key-value mechanism, where the query vector is derived from the visual feature vector, and the key and value vectors are derived from the non-visual feature vectors. This multi-head cross-attention mechanism adaptively learns the complementary relationships between features from different modalities, enhancing effective feature information. During model training, the cross-entropy loss function is used as the training loss, and the adaptive gradient descent optimizer is selected to adjust the model parameters. The training is iterated until the loss function value converges and the accuracy of the validation set is stable, resulting in a trained multimodal fusion encoder. This encoder can output the fused multimodal joint feature vector, which comprehensively represents the visual and non-visual feature information of the target motion.

[0083] S13. Select a training sample from the multimodal training dataset, extract the visual feature vector of the training sample through the visual feature extraction network, and calculate the standard joint feature vector of the training sample by combining the non-visual feature vector of the training sample with the trained multimodal fusion encoder.

[0084] Optionally, a training sample is randomly selected from the multimodal training dataset. First, the visual video data of this sample is used to extract features through the aforementioned pre-defined visual feature extraction network, obtaining the corresponding visual feature vector to ensure consistency in feature dimension and data distribution. Then, this visual feature vector, along with the non-visual feature vector from the same sample, is input into the trained multimodal fusion encoder. The non-visual feature vector undergoes the same preprocessing and feature extraction process as in the training phase to ensure input format matching. The multimodal fusion encoder performs multi-head cross-attention calculation on the two types of input feature vectors. Through internal feature transformation and fusion mechanisms, it outputs a fused multimodal joint feature vector. This vector serves as the standard joint feature vector, representing the true multimodal feature fusion result without visual perturbation, providing a benchmark reference for subsequent perturbation propagation analysis. The output dimension of the standard joint feature vector is pre-designed to match the input and output dimensions of the subsequent error perturbation simulator.

[0085] S14. Apply various artificial perturbations to the visual feature vectors to generate multiple perturbed visual feature vectors. Input each perturbed visual feature vector and the non-visual feature vector of the same sample into the trained multimodal fusion encoder to obtain the corresponding biased joint feature vector.

[0086] Optionally, various artificial perturbations are applied to the visual feature vectors to generate multiple perturbed visual feature vectors. Perturbation types include, but are not limited to, Gaussian perturbation, uniform perturbation, salt-and-pepper perturbation, and truncated perturbation. The parameter configurations for each perturbation are set according to the actual visual perception error scenario, simulating visual feature deviations caused by different interference factors such as motion blur, limb occlusion, and illumination changes. The application of artificial perturbations is achieved by adjusting the elements of the visual feature vectors dimensionally or in batches. Each perturbation type corresponds to a different adjustment logic. For example, Gaussian perturbation is achieved by adding perturbation values ​​conforming to a normal distribution to the feature vector elements, while uniform perturbation is achieved by randomly adjusting the feature vector element values ​​within a fixed range. After generating multiple visual feature vectors with different perturbation intensities and types, each perturbed visual feature vector is combined with the non-visual feature vector of the same sample and input into the trained multimodal fusion encoder. Because the input visual feature vector is perturbed, the joint feature vector output by the encoder will also be deviated accordingly, that is, the corresponding biased joint feature vector is obtained. Multiple biased joint feature vectors correspond to the feature fusion results under different visual perturbations, providing data support for analyzing the propagation law of visual perturbations to joint features.

[0087] S15. Calculate the first difference vector between the visual feature vector and each perturbed visual feature vector, and calculate the second difference vector between the standard joint feature vector and each biased joint feature vector.

[0088] Optionally, the first difference vector is the difference between the visual feature vector and each perturbed visual feature vector, calculated by element-wise subtraction. and the perturbated visual feature vector First difference vector: , where each element: , The first vector is the dimension index of the feature vector, which represents the degree and distribution of perturbation at the visual feature level. The second difference vector is the difference between the standard joint feature vector and each biased joint feature vector, where the standard joint feature vector is denoted as . The biased joint eigenvector is denoted as Then the second difference vector: Each element: , This serves as the dimensional index for the joint feature vector, representing the degree of deviation resulting from the propagation of visual perturbation to the joint feature level. During the calculation, it is ensured that the dimensions of the two difference vectors are consistent with their corresponding original feature vectors. Simultaneously, the validity of the difference vectors is verified, and outliers caused by calculation errors are removed to ensure that the difference vectors accurately reflect the correspondence between perturbation and deviation.

[0089] S16. Construct a differentiable model of a multilayer perceptron as an error perturbation simulator, take the first difference vector as input and the second difference vector as supervision target, train the error perturbation simulator by minimizing the mean square error loss function between the model prediction output and the supervision target, so that the error perturbation simulator approximates the propagation function from visual feature perturbation to joint feature bias, and obtain the preset differentiable error perturbation model after training.

[0090] Optionally, the error perturbation simulator is constructed using a multilayer perceptron, which is a differentiable model, ensuring that parameters can be adjusted via backpropagation during training. The model's input is the first difference vector, and its output is the predicted second difference vector. The supervised target is the true second difference vector. During training, the mean squared error loss function is used to measure the difference between the model's predicted output and the supervised target. The formula for the mean squared error loss function is: ,in, The loss value. Let be the dimension of the difference vector. The true second difference vector The numerical value of the dimension. The model predicts the output of the first... The numerical value of the dimension. During training, an adaptive momentum optimizer is used to update the model parameters. The optimizer adapts to the update requirements of different parameters by adjusting the learning rate, thus accelerating model convergence. During training, the changes in the loss values ​​of the training and validation sets are monitored in real time. When the loss value tends to stabilize and the loss value of the validation set no longer decreases, training is stopped. At this point, the model has approximated the propagation function from visual feature perturbation to joint feature bias. The error perturbation simulator after training is the pre-set differentiable error perturbation model. This model can accurately predict the drift of joint features based on the perturbation difference of visual features, providing reliable support for the subsequent calculation of error perception statistics.

[0091] In an optional embodiment, visual motion video data is input into a preset visual feature extraction network, and visual spatiotemporal feature vectors are output, including the following steps:

[0092] S21. Perform frame sampling and time alignment processing on the visual motion video data to obtain a standardized video frame sequence.

[0093] Optionally, a uniform sampling strategy is adopted for frame sampling. The sampling interval is calculated based on the preset target frame rate and the original video frame rate to ensure that the length of the sampled video frame sequence is consistent, avoiding feature extraction deviations caused by differences in video duration. Time alignment processing is implemented based on the timestamp information of the video frames. Using the timestamp of the first frame as a reference, the time synchronization calibration of the sampled frame sequence is performed to eliminate possible time offsets during acquisition. For video data with frame rate fluctuations, linear interpolation is used to supplement missing frames, ensuring the temporal continuity of the frame sequence. Standardization processing also includes unifying the size of the frame sequence, scaling all video frames to a preset fixed resolution to ensure a uniform frame sequence format input to the 3D convolutional neural network, providing standardized input for subsequent spatiotemporal feature extraction.

[0094] S22. Spatiotemporal features are extracted from the video frame sequence using a three-dimensional convolutional neural network to obtain a preliminary spatiotemporal feature map.

[0095] Optionally, the 3D convolutional neural network employs stacked 3D convolutional layers to achieve joint extraction of spatiotemporal features. Each convolutional layer consists of multiple 3D convolutional kernels, which simultaneously perform sliding convolution operations on the input video frame sequence in both spatial and temporal dimensions. In the spatial dimension, the convolutional kernels capture local spatial features within the video frame, such as the positions of human joints and limb contours; in the temporal dimension, the convolutional kernels capture motion change features between adjacent frames, such as limb movement trajectories and changes in the timing of actions. During the convolution operation, equal-length padding is used to maintain the feature map size consistent with the input, avoiding the loss of edge features. After multiple convolutional operations, the output is a preliminary spatiotemporal feature map containing spatiotemporal information at different levels. This feature map simultaneously encodes spatial structural information and temporal dynamic information.

[0096] S23. Perform pooling operation on the time dimension on the preliminary spatiotemporal feature map to obtain the temporal aggregated features.

[0097] Optionally, temporal pooling employs either temporal average pooling or temporal max pooling strategies. Based on the temporal distribution of the initial spatiotemporal feature map, a fixed-length time window is defined. For each feature value within a time window, average pooling calculates the mean of all feature values ​​within the window, while max pooling selects the maximum feature value within the window. This operation aggregates and compresses the features along the temporal dimension, preserving key motion information, reducing the dimensionality of the feature data, and decreasing subsequent computational load. The size and stride of the pooling window are preset based on the temporal scale characteristics of the action to ensure complete capture of key temporal features within an action cycle, avoiding loss of temporal information due to improper window settings. The final output is the aggregated temporal feature.

[0098] S24. Weighted spatiotemporal features are obtained by weighting the temporal aggregation features through a spatiotemporal attention layer.

[0099] Optionally, the spatiotemporal attention layer includes a spatial attention branch and a temporal attention branch, which respectively assign attention weights to the spatial and temporal dimensions of the temporally aggregated features. The spatial attention branch strengthens the feature responses of key areas of human motion and suppresses feature interference from irrelevant areas such as the background by calculating the importance weights of each spatial location in the feature map. The temporal attention branch assigns higher weights to features of key action sequence segments and weakens secondary features of the action transition phase by analyzing the contribution of the temporally aggregated features in different time segments. The weight calculation is based on the global statistical information of the features. The weight values ​​are mapped to between 0 and 1 using the sigmoid activation function. Then, the attention weights are multiplied element-wise with the temporally aggregated features to obtain the weighted spatiotemporal features, which are more focused on the key spatiotemporal information of the action.

[0100] S25. The weighted spatiotemporal features are mapped through a fully connected layer to obtain the visual spatiotemporal feature vector.

[0101] Optionally, the weighted spatiotemporal features are first flattened to convert the two-dimensional feature map into a one-dimensional feature vector, making it suitable for the input format of the fully connected layer. The fully connected layer consists of multiple hidden layers and an output layer. The hidden layers use non-linear activation functions to non-linearly transform the input one-dimensional feature vector, achieving high-order abstraction and dimensionality transformation of the features. Through multiple mappings, the high-dimensional feature vector is gradually compressed to a preset target dimension. The output layer uses a linear activation function to map the output of the last hidden layer to a fixed-dimensional visual spatiotemporal feature vector. This vector integrates key spatiotemporal feature information of the action, accurately representing the action characteristics of the target user during sports activities, providing reliable feature input for subsequent perturbation generation and error analysis.

[0102] In an optional embodiment, based on the visual spatiotemporal feature vector, a perturbation visual feature vector is generated through preset perturbation parameter configuration, including the following steps:

[0103] S31. Standardize the visual spatiotemporal feature vector to obtain the standard feature vector.

[0104] Optionally, the purpose of standardization is to eliminate dimensional differences between the dimensions of the visual spatiotemporal feature vector, ensuring that the numerical distribution of each dimension of the feature vector is on a uniform scale, and avoiding uneven perturbation effects caused by differences in the numerical ranges of different dimensions. Standardization employs the Z-Score standardization method, which calculates the mean and standard deviation based on global statistical information of the visual spatiotemporal feature vector. The specific formula is as follows: ,in For the standard eigenvector, the first The numerical value of the dimension. The original visual spatiotemporal feature vector is the first... The numerical value of the dimension. This represents the mean of all dimensions of the original visual spatiotemporal feature vector. This represents the standard deviation of all dimensions of the original visual spatiotemporal feature vector. This standardization process transforms the original visual spatiotemporal feature vector into a standard feature vector with a mean of 0 and a standard deviation of 1, providing a standardized basis for the subsequent generation and superposition of perturbation vectors.

[0105] S32. Generate multiple disturbance vectors according to the preset disturbance parameter configuration.

[0106] Optionally, the preset perturbation parameter configuration includes perturbation intensity parameters and perturbation direction matrix.

[0107] Optionally, the perturbation intensity parameter controls the magnitude of the perturbation. Its value is set based on the common amplitude range of visual perception errors in actual sports scenarios. By adjusting this parameter, different degrees of visual interference can be simulated, such as low-intensity perturbation corresponding to slight motion blur and high-intensity perturbation corresponding to severe limb occlusion. The perturbation direction matrix is ​​a two-dimensional matrix. The number of rows in the matrix is ​​consistent with the dimension of the standard feature vector, and the number of columns is consistent with the number of perturbation vectors to be generated. The elements in the matrix are used to define the perturbation direction of each perturbation vector in each feature dimension. Through this matrix, multi-directional perturbation coverage can be achieved to simulate the differences in feature deviation direction caused by different types of visual interference. When generating perturbation vectors, the direction of each perturbation vector is determined based on the perturbation direction matrix, and the perturbation amplitude in each dimension is determined in combination with the perturbation intensity parameter. Multiple perturbation vectors that conform to the preset parameter configuration are generated by random sampling to ensure that the distribution of perturbation vectors can fully cover potential visual perception error scenarios.

[0108] S33. Add each perturbation vector to the standard eigenvector to obtain multiple perturbation eigenvectors.

[0109] Optionally, since the standard feature vector and the perturbation vector have already achieved dimension matching through the aforementioned steps, the addition operation is performed element-wise. Specifically, the calculation logic is as follows: for the standard feature vector... and the perturbation vector The corresponding perturbation feature vector The The dimension value is: ,in, The dimension index of the feature vector. This represents the index of the perturbation vector. Through this addition operation, perturbations of different directions and intensities are superimposed onto the standardized feature vector, forming perturbation feature vectors containing different perturbation information. Each perturbation feature vector corresponds to a feature state under a simulated visual perception error scenario. During the addition process, the result needs to be validated to ensure that the superimposed feature values ​​do not overflow abnormally, thus guaranteeing the stability of subsequent processing.

[0110] S34. Perform inverse normalization on each perturbation feature vector to obtain multiple perturbation visual feature vectors.

[0111] Optionally, denormalization is the inverse process of normalization. Its purpose is to restore the normalized feature vector after superimposed perturbation to the original visual spatiotemporal feature vector, ensuring that the generated perturbed visual feature vector has consistent data distribution characteristics with the original visual spatiotemporal feature vector, facilitating subsequent calculation of difference features and error analysis. The formula for denormalization is: ,in, For the first The perturbation visual feature vector of the first The numerical value of the dimension. For the first The perturbation feature vector of the th _i_ ... The numerical value of the dimension. and These represent the mean and standard deviation of the original visual spatiotemporal feature vectors, respectively. Through this denormalization process, multiple perturbed visual feature vectors with the same dimensions as the original visual spatiotemporal feature vectors are finally obtained, providing suitable input data for the calculation of difference features in subsequent steps.

[0112] In an optional embodiment, the difference between the visual spatiotemporal feature vector and the perturbed visual feature vector is input into a preset differentiable error perturbation model for processing to obtain multiple predicted joint feature drift vectors, including the following steps:

[0113] S41. Calculate the difference feature between the visual spatiotemporal feature vector and each perturbation visual feature vector.

[0114] Optionally, the difference features are calculated using element-wise subtraction. For each perturbation visual feature vector, the difference in the corresponding dimension is calculated between it and the original visual spatiotemporal feature vector. Let the visual spatiotemporal feature vector be... , No. The perturbation visual feature vectors are The corresponding difference feature The The dimension value is: ,in, The dimension index of the feature vector. This represents the index of the perturbation visual feature vector. This calculation method accurately captures the degree and distribution of visual feature deviations under each perturbation scenario. Each difference feature corresponds to a feature deviation state caused by a visual perception error, providing differentiated deviation feature data for subsequent model input. After calculation, all difference features are validated, and those with dimension mismatches or numerical anomalies are removed to ensure the data quality input to the model.

[0115] S42. Input each difference feature into the preset differentiable error perturbation model to obtain the predicted joint feature drift vector.

[0116] Optionally, the preset differentiable error perturbation model is a trained multilayer perceptron model that has learned the propagation mapping relationship from visual feature perturbation differences to joint feature biases. During input, it is necessary to ensure that the dimension of the difference features is consistent with the dimension of the model input layer. If there is a dimension mismatch, the dimension of the difference features is transformed through a dimension adaptive adjustment module to ensure input format adaptation. After receiving the difference features, the model maps the bias at the visual feature level to the joint feature level through internal multilayer nonlinear mapping operations, outputting the corresponding predicted joint feature drift vector. This vector represents the possible drift state of the multimodal joint features under the corresponding visual perturbation, providing core data support for the subsequent calculation of error perception statistics.

[0117] S43. Normalize each predicted joint feature drift vector to obtain a normalized joint feature drift vector.

[0118] Optionally, the purpose of normalization is to eliminate the dimensional differences between different joint feature drift vectors, ensuring that all drift vectors are on a uniform numerical scale, thus avoiding uneven scaling effects caused by differences in the numerical range of the original drift vectors. The normalization process uses the Min-Max normalization method, with the specific formula as follows: ,in, For the first The normalized joint feature drift vector of the first The numerical value of the dimension. For the first The original predicted joint feature drift vector is the first... The numerical value of the dimension. For the first The minimum value among all dimensions of the original predicted joint feature drift vector. For the first The maximum value across all dimensions of the original predicted joint feature drift vector is used for normalization. This normalization process maps all original predicted joint feature drift vectors to a fixed numerical range, providing a standardized basis for subsequent scaling.

[0119] S44. Based on the preset drift intensity coefficient, scale each normalized joint feature drift vector to obtain multiple predicted joint feature drift vectors.

[0120] Optionally, the preset drift intensity coefficient is a parameter set according to the requirements of actual sports movement error correction scenarios. Its value is determined based on the movement sensitivity and error tolerance of different sports types, and is used to adjust the intensity level of the joint feature drift vector so that the final output drift vector better matches the error representation requirements of the actual application scenario. Scaling is implemented using element-wise multiplication. The specific calculation logic is as follows: for the... Normalized joint feature drift vector and preset drift intensity coefficient Scaled joint feature drift vector The The dimension value is: This scaling process allows for the enhancement or weakening of the representational strength of joint feature shifts according to actual needs, ultimately yielding multiple normalized and scaled predicted joint feature shift vectors. This ensures that the vectors can accurately quantify the degree of joint feature shifts under different visual perturbations.

[0121] In an optional embodiment, an error-aware statistic is calculated based on the predicted joint feature drift vectors, including the following steps:

[0122] S51. Calculate the covariance matrix of multiple joint feature drift vectors to obtain the drift covariance matrix.

[0123] Optionally, the covariance matrix is ​​used to characterize the degree of linear correlation among the dimensions of multiple joint feature drift vectors, and its calculation is based on the statistical properties of the joint feature drift vectors. First, multiple joint feature drift vectors are constructed into a feature matrix, assuming a total number of features. There are joint feature drift vectors, each vector having a dimension of . Then the characteristic matrix n rows A matrix of columns, the first column in the matrix Line 1 The element of the column is the first The value of the j-th dimension of the joint feature drift vector. The covariance matrix is ​​calculated first by considering the feature matrix. The matrix is ​​centered by subtracting the mean of the corresponding column from each element. Then the drift covariance matrix is ​​calculated. The calculation method is to multiply the transpose of the feature matrix and the centered matrix, and then divide by the number of samples minus one. The specific formula is as follows: ,in, for OK The drift covariance matrix of the column, For a centralized matrix transpose, This represents the number of joint feature drift vectors. The drift covariance matrix obtained through this calculation can comprehensively reflect the correlation characteristics between the dimensions of the joint feature drift vectors, providing basic data for subsequent eigenvalue decomposition.

[0124] S52. Perform eigenvalue decomposition on the drift covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors.

[0125] Optionally, eigenvalue decomposition is the process of decomposing a real symmetric matrix into eigenvalues ​​and eigenvectors. Since the drift covariance matrix is ​​a real symmetric matrix, eigenvalue decomposition can be achieved through orthogonal diagonalization. Specifically, the decomposition process involves finding a set of orthogonal vectors. and a set of real numbers This makes the drift covariance matrix satisfy: ,in, For the first 1 eigenvalue, To and The corresponding eigenvectors. After eigenvalue decomposition, the eigenvalues ​​are sorted in descending order, and the corresponding eigenvectors are rearranged accordingly. The magnitude of the eigenvalue reflects the variance in the direction of the corresponding eigenvector, that is, the strength of the joint feature shift in that direction; the eigenvector represents the main direction of the joint feature shift. Eigenvalue decomposition can extract key information about the joint feature shift, laying the foundation for the calculation of error-aware statistics.

[0126] S53. Based on the eigenvalues, the error perception statistic is obtained; whereby the expression for the error perception statistic is:

[0127]

[0128] In the formula, This represents the error perception statistic. For the first 1 eigenvalue, Represents the total number of eigenvalues. It is a very small constant.

[0129] The meanings of the parameters in the formula are as follows: This represents the error perception statistic, which quantifies the overall uncertainty of the joint feature drift vector by logarithmic weighting of the eigenvalues. The larger the value, the worse the stability of the joint features under perturbation and the more significant the impact of visual perception error. For the first The eigenvalues, that is, the ordered eigenvalues ​​obtained by eigenvalue decomposition, reflect the strength of the joint feature drift in the corresponding direction; This represents the total number of eigenvalues, consistent with the dimension of the joint feature drift vector; This is a local constant; the purpose of setting this constant is to avoid the situation where the eigenvalues ​​approach 0. To prevent the occurrence of meaningless negative infinity, we must ensure the validity and stability of the statistical calculations. The value of needs to be much smaller than the order of magnitude of the smallest eigenvalue, while not affecting the overall representational effect of the statistic. During calculation, first consider each eigenvalue... calculate and Multiply the products, sum all the products, and finally add a minimal constant. The final error perception statistic is obtained. This statistic can accurately characterize the stability of visual spatiotemporal feature vectors under perturbation, providing a reliable basis for error assessment for subsequent feature fusion and error correction instruction generation.

[0130] In an optional embodiment, the visual spatiotemporal feature vector and the error perception statistics are fused to generate a fused feature vector, and an error correction instruction is generated based on the fused feature vector, including the following steps:

[0131] S61. Extend the error perception statistics to an error perception vector with the same dimension as the visual spatiotemporal feature vector.

[0132] Optionally, since the error-perceived statistics are single scalar values ​​while the visual spatiotemporal feature vectors are high-dimensional vectors, to achieve element-wise fusion, the scalar error-perceived statistics need to be converted into vectors of the same dimension through a dimension expansion operation. This expansion is implemented using a broadcast mechanism, that is, based on the dimensionality information of the visual spatiotemporal feature vectors, the error-perceived statistics are... Repeatedly padding the vector with the same dimension as the visual spatiotemporal feature vector yields the error-aware vector. Let the dimension of the visual spatiotemporal feature vector be... Then the error perception vector The The dimension value is: This extended operation ensures that error-aware information is applied evenly to each dimension of the visual spatiotemporal feature vector, providing dimension-matched input data for subsequent element-by-element fusion, while fully preserving the visual feature stability information represented by the error-aware statistics.

[0133] S62. The visual spatiotemporal feature vector and the error perception vector are fused by element-wise multiplication to obtain a weighted visual feature vector.

[0134] Optionally, the core logic of element-wise multiplication fusion is to dynamically weight each dimension of the visual spatiotemporal feature vector using the error-aware vector, thereby achieving differentiated adjustments for feature dimensions affected by different degrees of error. Let the visual spatiotemporal feature vector be... The error sensing vector is Then the weighted visual feature vector The The dimension value is: Since the value of the error-perceived vector is derived from the error-perceived statistics, and a larger value of the error-perceived statistics indicates poorer stability of the visual spatiotemporal feature vector, this element-wise multiplication can weaken the feature dimensions with poor stability (when...). When it is large, if For effective features, although the product result is amplified, it can still be distinguished by combining it with subsequent classification logic; if To address the issue of features susceptible to error interference (making the anomalies of the product results easier to identify), we relatively strengthen the feature dimensions with good stability, thereby improving the reliability of the fused features and highlighting the contribution of effective feature information to action error classification.

[0135] S63. Input the weighted visual feature vector into the preset action classifier to obtain the probability distribution of action error categories.

[0136] Optionally, the preset action classifier employs a deep neural network architecture, consisting of multiple fully connected layers and an output layer. The hidden layers use non-linear activation functions to achieve high-order feature abstraction, and the output layer uses a softmax activation function to map the network output to the probability values ​​corresponding to each action error category. The training process of the action classifier is based on a multimodal training dataset labeled with action error category labels. The training objective is to minimize the cross-entropy loss between the predicted probability distribution and the true labels (using one-hot encoding). After inputting a weighted visual feature vector, the classifier performs layer-by-layer transformation and abstraction of the features through internal forward propagation operations, ultimately outputting a probability distribution vector with the same dimension as the preset number of action error categories. ,in This indicates that the current action belongs to the first... The probability of a class of error is given, and the sum of all probability values ​​is 1, which comprehensively represents the possibility that the current action belongs to a class of error.

[0137] S64. Based on the probability distribution of action error categories, select the action error category with the highest probability.

[0138] Optionally, the error category with the highest confidence level is selected from the probability distribution vector and used as the core error judgment result for the current action. The specific implementation logic involves iterating through the probability distribution vector. Given all elements, find the index of the element with the largest value. ,Right now , , The index represents the total number of action error categories. The corresponding action error category is the main error category of the current action. To improve the reliability of the classification results, a probability threshold filtering mechanism can be set. If the highest probability value is lower than the preset threshold, the current action error feature is determined to be unclear, and visual motion video data needs to be collected again for analysis; if it is higher than or equal to the threshold, the error category is confirmed as a valid judgment result, and the process proceeds to the subsequent error correction instruction generation stage.

[0139] S65. Based on the error category, query the preset error correction instruction library and generate the corresponding error correction instruction.

[0140] Optionally, the error correction instructions include an error description and adjustment suggestions.

[0141] Optionally, the preset error correction instruction library is a structured database that stores the mapping relationship between various action error categories and their corresponding error correction instructions. This mapping relationship is constructed through expert experience and validation using a large amount of training data. The core structure of the error correction instruction library includes three main fields: error category identifier, error description, and adjustment suggestion. The error category identifier corresponds one-to-one with the error category output by the action classifier. The error description is used to accurately define the specific manifestation of the erroneous action, and the adjustment suggestion is used to provide targeted action correction solutions. When generating an error correction instruction, the system first performs a matching query in the error correction instruction library based on the determined action error category to obtain the corresponding error description and adjustment suggestion fields. Then, the two types of information are concatenated according to a preset format to form a complete error correction instruction. The error description clearly describes the core characteristics and location of the current action error, while the adjustment suggestion clarifies the direction, key points, and specific requirements of the action adjustment, ensuring that the user can accurately understand the error and complete the targeted adjustment.

[0142] The aforementioned computer vision-based sports motion correction method first acquires visual motion video data of the target user's sports activities. It then uses a visual feature extraction network containing a 3D convolutional neural network, a spatiotemporal attention layer, and a fully connected layer to extract spatiotemporal visual feature vectors. Next, based on preset perturbation parameters, it generates perturbed visual feature vectors from these spatiotemporal feature vectors. The difference between the two is input into a preset differentiable error perturbation model to obtain multiple predicted joint feature drift vectors. Subsequently, based on these vectors, it calculates error perception statistics characterizing the stability of the perturbation of the spatiotemporal feature vectors. Finally, it fuses the spatiotemporal feature vectors and error perception statistics to generate a fused feature vector, and uses this to generate correction instructions. This coherent approach effectively considers visual perception errors, improves the accuracy and reliability of sports motion correction, and ensures that the generated correction instructions accurately guide users to adjust their sports movements.

[0143] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0144] Based on the same inventive concept, this application also provides a computer vision-based sports motion correction device for implementing the aforementioned computer vision-based sports motion correction method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more computer vision-based sports motion correction device embodiments provided below can be found in the limitations of the computer vision-based sports motion correction method described above, and will not be repeated here.

[0145] In one exemplary embodiment, such as Figure 3 As shown, a schematic diagram of a computer vision-based sports motion correction device 20 is provided, comprising:

[0146] The data acquisition and feature extraction module 21 is used to acquire visual motion video data of the target user during sports activities, and input the visual motion video data into a preset visual feature extraction network to output a visual spatiotemporal feature vector; the visual feature extraction network includes a three-dimensional convolutional neural network, a spatiotemporal attention layer and a fully connected layer;

[0147] The perturbation generation module 22 is used to generate a perturbation visual feature vector based on the visual spatiotemporal feature vector and through preset perturbation parameter configuration.

[0148] The perturbation error analysis module 23 is used to input the difference features between the visual spatiotemporal feature vector and the perturbation visual feature vector into the preset differentiable error perturbation model for processing, so as to obtain multiple predicted joint feature drift vectors.

[0149] Error calculation module 24 is used to calculate error perception statistics based on multiple predicted joint feature drift vectors; wherein, error perception statistics are used to characterize the stability of visual spatiotemporal feature vectors under perturbation;

[0150] The error correction instruction generation module 25 is used to fuse the visual spatiotemporal feature vector and the error perception statistics to generate a fused feature vector, and to generate error correction instructions based on the fused feature vector; wherein, the error correction instructions are used to instruct the user to adjust the sports action.

[0151] Furthermore, the pre-defined differentiable error perturbation model is obtained through the following method:

[0152] S11. Obtain a multimodal training dataset consisting of visual video data, inertial measurement unit data, and electromyographic signal data that are strictly synchronized with time stamps; each sample in the multimodal training dataset is associated with an expert-annotated action error category label;

[0153] S12. Using the multimodal training dataset, train a multimodal fusion encoder to obtain the trained multimodal fusion encoder. The multimodal fusion encoder takes visual feature vectors and non-visual feature vectors as inputs, performs feature fusion through an internal multi-head cross-attention mechanism, and outputs the fused multimodal joint feature vector.

[0154] S13. Select a training sample from the multimodal training dataset, extract the visual feature vector of the training sample through the visual feature extraction network, and calculate the standard joint feature vector of the training sample by combining the non-visual feature vector of the training sample with the trained multimodal fusion encoder.

[0155] S14. Apply various artificial perturbations to the visual feature vectors to generate multiple perturbed visual feature vectors. Input each perturbed visual feature vector and the non-visual feature vector of the same sample into the trained multimodal fusion encoder to obtain the corresponding biased joint feature vector.

[0156] S15. Calculate the first difference vector between the visual feature vector and each perturbed visual feature vector, and calculate the second difference vector between the standard joint feature vector and each biased joint feature vector.

[0157] S16. Construct a differentiable model of a multilayer perceptron as an error perturbation simulator, take the first difference vector as input and the second difference vector as supervision target, train the error perturbation simulator by minimizing the mean square error loss function between the model prediction output and the supervision target, so that the error perturbation simulator approximates the propagation function from visual feature perturbation to joint feature bias, and obtain the preset differentiable error perturbation model after training.

[0158] Furthermore, the data acquisition and feature extraction module 21 is also used for:

[0159] S21. Perform frame sampling and time alignment processing on the visual motion video data to obtain a standardized video frame sequence;

[0160] S22. Spatiotemporal feature extraction is performed on the video frame sequence using a three-dimensional convolutional neural network to obtain a preliminary spatiotemporal feature map;

[0161] S23. Perform pooling operation on the time dimension on the preliminary spatiotemporal feature map to obtain the temporal aggregated features;

[0162] S24. Weighted spatiotemporal features are obtained by weighting the temporal aggregation features through a spatiotemporal attention layer.

[0163] S25. The weighted spatiotemporal features are mapped through a fully connected layer to obtain the visual spatiotemporal feature vector.

[0164] Furthermore, the disturbance generation module 22 is also used for:

[0165] S31. Standardize the visual spatiotemporal feature vector to obtain the standard feature vector;

[0166] S32. Generate multiple disturbance vectors according to the preset disturbance parameter configuration; the preset disturbance parameter configuration includes disturbance intensity parameters and disturbance direction matrix;

[0167] S33. Add each perturbation vector to the standard eigenvector to obtain multiple perturbation eigenvectors;

[0168] S34. Perform inverse normalization on each perturbation feature vector to obtain multiple perturbation visual feature vectors.

[0169] Furthermore, the disturbance error analysis module 23 is also used for:

[0170] S41. Calculate the difference feature between the visual spatiotemporal feature vector and each perturbation visual feature vector;

[0171] S42. Input each difference feature into the preset differentiable error perturbation model to obtain the predicted joint feature drift vector;

[0172] S43. Normalize each predicted joint feature drift vector to obtain a normalized joint feature drift vector.

[0173] S44. Based on the preset drift intensity coefficient, scale each normalized joint feature drift vector to obtain multiple predicted joint feature drift vectors.

[0174] Furthermore, the error calculation module 24 is also used for:

[0175] S51. Calculate the covariance matrix of multiple joint feature drift vectors to obtain the drift covariance matrix;

[0176] S52. Perform eigenvalue decomposition on the drift covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors;

[0177] S53. Based on the eigenvalues, the error perception statistic is obtained; whereby the expression for the error perception statistic is:

[0178]

[0179] In the formula, This represents the error perception statistic. For the first 1 eigenvalue, Represents the total number of eigenvalues. It is a very small constant.

[0180] Furthermore, the error correction instruction generation module 25 is also used for:

[0181] S61. Extend the error perception statistics to an error perception vector with the same dimension as the visual spatiotemporal feature vector.

[0182] S62. The visual spatiotemporal feature vector and the error perception vector are fused by element-wise multiplication to obtain a weighted visual feature vector.

[0183] S63. Input the weighted visual feature vector into the preset action classifier to obtain the probability distribution of action error categories;

[0184] S64. Based on the probability distribution of action error categories, select the action error category with the highest probability;

[0185] S65. Based on the action error category, query the preset error correction instruction library and generate the corresponding error correction instruction; wherein, the error correction instruction includes an error description and adjustment suggestions.

[0186] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the computer vision-based sports motion correction method as described above.

[0187] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0188] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0189] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A computer vision-based method for correcting errors in sports movements, characterized in that, The method includes: S1. Acquire visual motion video data of the target user during sports activities, and input the visual motion video data into a preset visual feature extraction network to output a visual spatiotemporal feature vector; the visual feature extraction network includes a three-dimensional convolutional neural network, a spatiotemporal attention layer, and a fully connected layer; S2. Based on the visual spatiotemporal feature vector, a perturbation visual feature vector is generated through preset perturbation parameter configuration; S3. Input the difference feature between the visual spatiotemporal feature vector and the perturbation visual feature vector into a preset differentiable error perturbation model for processing to obtain multiple predicted joint feature drift vectors. S4. Based on the predicted multiple joint feature drift vectors, calculate the error perception statistic; wherein, the error perception statistic is used to characterize the stability of the visual spatiotemporal feature vector under perturbation; S5. The visual spatiotemporal feature vector and the error perception statistics are fused to generate a fused feature vector, and an error correction instruction is generated based on the fused feature vector; wherein, the error correction instruction is used to instruct the user to adjust the sports action.

2. The method according to claim 1, characterized in that, The preset differentiable error perturbation model is obtained through the following method: S11. Obtain a multimodal training dataset consisting of visual video data, inertial measurement unit data, and electromyographic signal data that are strictly synchronized with time stamps; each sample in the multimodal training dataset is associated with an expert-annotated action error category label; S12. Using the multimodal training dataset, train a multimodal fusion encoder to obtain a trained multimodal fusion encoder; wherein, the multimodal fusion encoder takes visual feature vectors and non-visual feature vectors as input, performs feature fusion through an internal multi-head cross-attention mechanism, and outputs a fused multimodal joint feature vector; S13. Select a training sample from the multimodal training dataset, extract the visual feature vector of the training sample through the visual feature extraction network, and calculate the standard joint feature vector of the training sample by combining the non-visual feature vector of the training sample with the trained multimodal fusion encoder. S14. Apply various artificial perturbations to the visual feature vector to generate multiple perturbed visual feature vectors, and input each perturbed visual feature vector and the non-visual feature vector of the same sample into the trained multimodal fusion encoder to obtain the corresponding biased joint feature vector. S15. Calculate the first difference vector between the visual feature vector and each of the perturbed visual feature vectors, and calculate the second difference vector between the standard joint feature vector and each of the biased joint feature vectors; S16. Construct a differentiable model of a multilayer perceptron as an error perturbation simulator, take the first difference vector as input, take the second difference vector as supervision target, train the error perturbation simulator by minimizing the mean squared error loss function between the model prediction output and the supervision target, so that the error perturbation simulator approximates the propagation function from visual feature perturbation to joint feature bias, and obtain the preset differentiable error perturbation model after training.

3. The method according to claim 1, characterized in that, The step of inputting the visual motion video data into a preset visual feature extraction network and outputting a visual spatiotemporal feature vector includes: S21. Perform frame sampling and time alignment processing on the visual motion video data to obtain a standardized video frame sequence; S22. The video frame sequence is subjected to spatiotemporal feature extraction using the three-dimensional convolutional neural network to obtain a preliminary spatiotemporal feature map; S23. Perform a pooling operation on the time dimension on the preliminary spatiotemporal feature map to obtain the temporal aggregated feature; S24. The time aggregation features are weighted through the spatiotemporal attention layer to obtain weighted spatiotemporal features; S25. The weighted spatiotemporal features are mapped through the fully connected layer to obtain the visual spatiotemporal feature vector.

4. The method according to claim 1, characterized in that, The step of generating a perturbed visual feature vector based on the visual spatiotemporal feature vector and through preset perturbation parameter configuration includes: S31. Standardize the visual spatiotemporal feature vector to obtain a standard feature vector; S32. Generate multiple disturbance vectors according to the preset disturbance parameter configuration; the preset disturbance parameter configuration includes disturbance intensity parameters and disturbance direction matrix; S33. Add each of the perturbation vectors to the standard feature vector to obtain multiple perturbation feature vectors; S34. Perform inverse normalization on each of the perturbation feature vectors to obtain multiple perturbation visual feature vectors.

5. The method according to claim 2, characterized in that, The difference between the visual spatiotemporal feature vector and the perturbed visual feature vector is input into a preset differentiable error perturbation model for processing to obtain multiple predicted joint feature drift vectors, including: S41. Calculate the difference feature between the visual spatiotemporal feature vector and each of the perturbation visual feature vectors; S42. Input each difference feature into the preset differentiable error perturbation model to obtain the predicted joint feature drift vector; S43. Normalize each predicted joint feature drift vector to obtain a normalized joint feature drift vector. S44. Based on a preset drift intensity coefficient, scale each of the normalized joint feature drift vectors to obtain the predicted multiple joint feature drift vectors.

6. The method according to claim 1, characterized in that, The calculation of error-aware statistics based on the predicted joint feature drift vectors includes: S51. Calculate the covariance matrix of the multiple joint feature drift vectors to obtain the drift covariance matrix; S52. Perform eigenvalue decomposition on the drift covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors; S53. Based on the feature values, the error-aware statistic is obtained; wherein, the expression of the error-aware statistic is: In the formula, This represents the error perception statistic. For the first 1 eigenvalue, Represents the total number of eigenvalues. It is a very small constant.

7. The method according to claim 1, characterized in that, The step of fusing the visual spatiotemporal feature vector and the error perception statistics to generate a fused feature vector, and generating an error correction instruction based on the fused feature vector, includes: S61. Expand the error perception statistics into an error perception vector with the same dimension as the visual spatiotemporal feature vector. S62. The visual spatiotemporal feature vector and the error perception vector are fused by element-wise multiplication to obtain a weighted visual feature vector; S63. Input the weighted visual feature vector into a preset action classifier to obtain the probability distribution of action error categories; S64. Based on the probability distribution of the action error categories, select the action error category with the highest probability; S65. Based on the error category of the action, query the preset error correction instruction library and generate the corresponding error correction instruction; wherein, the error correction instruction includes an error description and adjustment suggestions.

8. A sports motion error correction device based on computer vision, characterized in that, The device includes: The data acquisition and feature extraction module is used to acquire visual motion video data of the target user during sports activities, and input the visual motion video data into a preset visual feature extraction network to output a visual spatiotemporal feature vector; the visual feature extraction network includes a three-dimensional convolutional neural network, a spatiotemporal attention layer and a fully connected layer; The perturbation generation module is used to generate a perturbation visual feature vector based on the visual spatiotemporal feature vector and through preset perturbation parameter configuration. The perturbation error analysis module is used to input the difference features between the visual spatiotemporal feature vector and the perturbation visual feature vector into a preset differentiable error perturbation model for processing, so as to obtain multiple predicted joint feature drift vectors. An error calculation module is used to calculate an error-perceived statistic based on the predicted multiple joint feature drift vectors; wherein the error-perceived statistic is used to characterize the stability of the visual spatiotemporal feature vector under perturbation; The error correction instruction generation module is used to fuse the visual spatiotemporal feature vector and the error perception statistics to generate a fused feature vector, and to generate an error correction instruction based on the fused feature vector; wherein, the error correction instruction is used to instruct the user to adjust the sports action.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images