A dance motion trajectory correction evaluation method based on three-dimensional motion capture

Abstract

The present application relates to the technical field of motion capture, in particular to a dance motion trajectory error correction evaluation method based on three-dimensional motion capture, which is realized based on a multi-modal edge collaborative motion capture system and comprises the following steps: color-depth image sequences and limb inertial motion data of a student's dance motion are synchronously collected through multiple synchronously calibrated depth cameras and wearable inertial measurement units, and a Bayesian multi-modal adaptive fusion algorithm is used to generate an initial spatiotemporal trajectory sequence of the student's three-dimensional skeleton joint; the generated initial spatiotemporal trajectory sequence of the student's three-dimensional skeleton joint is input into a spatiotemporal decoupling graph convolution network. The present application has the advantages of objective quantitative evaluation, real-time feedback, fine-grained error location and cause positioning, and robustness against occlusion interference, and is suitable for online or offline auxiliary teaching scenes in the field of dance education and training, which can effectively reduce the burden of teachers and improve the efficiency of student motion correction.

Description

Technical Field

[0001] This invention relates to the field of motion capture technology, specifically to a method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture. Background Technology

[0002] In the field of dance education and training, traditional movement correction assessment mainly relies on the coach's visual observation and verbal guidance. The assessment results are significantly affected by the coach's personal experience, fatigue, and subjective preferences, and lack objective and reproducible quantitative indicators. Different teachers often disagree on the identification of the same error.

[0003] However, with the development of motion capture technology, automated evaluation schemes based on optical markers or wearable inertial sensors have emerged. While optical motion capture systems offer high accuracy, they are expensive, require specialized calibration spaces, and are prone to losing their markers and causing trajectory breaks when dancers experience severe self-occlusion or cluttered backgrounds. Inertial sensor solutions offer strong wearability but suffer from cumulative drift errors and insufficient absolute spatial positioning accuracy; inertial data alone cannot accurately describe the spatial positional deviation of joint trajectories. Furthermore, existing automated evaluation methods often employ dynamic time warping algorithms or manual feature template matching, only outputting an overall similarity score. They cannot precisely locate the deviation direction and offset distance of specific joints in specific motion phases, nor can they distinguish whether the error is caused by joint trajectory deviation, incorrect center of gravity transfer, or temporal phase misalignment. They also lack effective compensation mechanisms for missing trajectories, leading to a significant decrease in the reliability of evaluations in complex scenarios. These shortcomings limit the real-time performance and accuracy of dance-assisted teaching systems in practical teaching scenarios.

[0004] Based on this, the present invention provides a method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture to solve the aforementioned technical problems. Summary of the Invention

[0005] The purpose of this invention is to provide a method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture, thereby solving the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention proposes a dance motion trajectory error correction and evaluation method based on three-dimensional motion capture. The method is implemented based on a multimodal edge collaborative motion capture system and includes the following steps:

[0008] S1: Using multiple synchronously calibrated depth cameras and wearable inertial measurement units, color-depth image sequences and limb inertial motion data of the student's dance movements are collected synchronously. The Bayesian multimodal adaptive fusion algorithm is used to generate the initial spatiotemporal trajectory sequence of the student's three-dimensional skeleton joints.

[0009] S2: Input the initial spatiotemporal trajectory sequence of the student's 3D skeleton joints generated in S1 into the spatiotemporal decoupling graph convolutional network, extract the joint local spatial dependence features and global temporal evolution features of the student's movements, and perform real-time physical consistency correction of joint angles and bone length through the pre-kinematic constraint network, and output a physically consistent student movement trajectory sequence.

[0010] S3: Obtain the standard three-dimensional skeleton trajectory sequence of the teacher's standard dance movements. Using the rhythm density curve extracted from the physically consistent student movement trajectory sequence output by S2 as a condition, use the adaptive rhythm dynamic trajectory template generator to perform non-linear time scaling on the teacher's standard three-dimensional skeleton trajectory sequence to generate an adaptive standard trajectory template that is aligned with the rhythm of the student's movements.

[0011] S4: Input the physically consistent trainee motion trajectory sequence output by S2 and the adaptive standard trajectory template output by S3 into the multi-scale trajectory deviation decoupling module. Use the phase-sensitive dynamic trajectory warping algorithm to calculate the three-dimensional deviation vector of each skeleton joint point in each time phase, form a spatiotemporal deviation vector field, and decouple the joint trajectory deviation characteristics, center of gravity transfer deviation characteristics and temporal offset characteristics from it.

[0012] S5: Construct a heterogeneous graph attention error diagnosis network. The spatiotemporal deviation vector field output by S4 and the decoupled joint trajectory deviation features, center of gravity transfer deviation features and temporal offset features are used as input to this network. The output is the spatial localization of the faulty joint, the error type classification result and the error cause attribution label. Fine-grained error correction prompts containing specific joint names, deviation directions and quantized offset distances are generated.

[0013] S6: Perform joint trajectory continuity detection on the physically consistent trainee motion trajectory sequence output by S2. When a missing frame segment of the trajectory is identified due to self-occlusion or complex dynamic background, the conditional temporal trajectory reconstruction module is activated. Using the physically consistent joint trajectory in the neighborhood of the missing segment in S2 and the spatiotemporal deviation vector field of the corresponding time window in S4 as joint conditions, a complete sequence of missing joint trajectories is generated. The completed trainee motion trajectory sequence is then re-input into S5 for error diagnosis, forming a closed-loop error correction evaluation.

[0014] Preferably, the implementation process of the Bayesian multimodal adaptive fusion algorithm in step S1 is as follows:

[0015] S1.1: Perform frame-by-frame human body region segmentation on color-depth image sequences acquired by multiple depth cameras, regress two-dimensional heatmaps of each joint point using a deep convolutional neural network, and reconstruct the three-dimensional joint point positions through multi-view epipolar geometric triangulation.

[0016] S1.2: Perform attitude calculation on the acceleration, angular velocity and geomagnetic data collected by the wearable inertial measurement unit to obtain the spatial orientation of each limb and the relative rotation quaternion of the joints, and calculate the coordinates of the three-dimensional joints based on the forward kinematics of the human kinetic chain.

[0017] S1.3: Using the visual 3D joint coordinates obtained in S1.1 and the inertial 3D joint coordinates obtained in S1.2 as observations, construct a Bayesian fusion filter based on dynamic confidence weighting, estimate the covariance of observation noise for each modality by jointly estimating the visual reprojection error and the inertial sensor residual, and output the fused initial spatiotemporal trajectory sequence of the student's 3D skeleton joints.

[0018] Preferably, the implementation process of the spatiotemporal decoupling graph convolutional network and the preceding kinematic constraint network in step S2 is as follows:

[0019] S2.1: Construct a spatiotemporally decoupled graph convolutional network, which consists of a spatial graph convolutional branch and a temporal convolutional branch in parallel. The spatial graph convolutional branch uses the natural connection relationship of the human skeleton as a basis and adds a learnable adaptive adjacency matrix to encode the spatial coordinates of the joints in each frame of the initial spatiotemporal trajectory sequence output by S1. The temporal convolutional branch uses dilated temporal convolution to perform temporal modeling of the spatial features of the joints in multiple consecutive frames. The output features of the two branches are merged through a gating fusion mechanism to obtain the spatiotemporal features of the trainee's actions.

[0020] S2.2: Input the spatiotemporal features of the trainee’s movements into the pre-kinematic constraint network. This network includes a bone length constraint layer and a joint angle limiting layer. The bone length constraint layer corrects trajectory distortion by forcing the length of each bone segment to be consistent with the trainee’s statically calibrated bone length. The joint angle limiting layer projects and corrects joint angles that exceed physiological limits based on the human joint range of motion library.

[0021] S2.3: The pre-kinematic constraint network outputs a physically consistent sequence of student movement trajectories, in which the coordinates of each joint point satisfy the constraints of constant bone length and physiological limits of joint angles.

[0022] Preferably, the implementation process of the adaptive rhythm dynamic trajectory template generator in step S3 is as follows:

[0023] S3.1: Extract rhythm features from the physically consistent trainee motion trajectory sequence output by S2, calculate the peak point sequence of the combined velocity of each joint of the trainee, and generate the trainee rhythm density curve through kernel density estimation.

[0024] S3.2: Generate teacher rhythm density curves using the same rhythm feature extraction method for the teacher's standard three-dimensional skeleton trajectory sequence;

[0025] S3.3: Using the student rhythm density curve as the target distribution and the teacher rhythm density curve as the source distribution, the nonlinear monotonic transformation function of the time axis is solved using the one-dimensional optimal transport mapping. This transformation function is then applied to the time axis of the teacher's standard three-dimensional skeleton trajectory sequence to generate an adaptive standard trajectory template after time retargeting.

[0026] Preferably, the implementation process of the phase-sensitive dynamic trajectory straightening algorithm in step S4 is as follows:

[0027] S4.1: Using the physically consistent trainee motion trajectory sequence output by S2 and the adaptive standard trajectory template output by S3 as input, construct the trajectory distance matrix for each joint. The distance metric adopts a composite distance function that combines the Euclidean distance of the joint space and the motion phase.

[0028] S4.2: Apply bone length preservation constraints and joint point local rigidity constraints to all relevant nodes, and solve the globally optimal time curvature path through multi-sequence synchronous dynamic time warping to ensure that the time curvature paths of different joint points satisfy the consistency of spatial structure.

[0029] S4.3: Based on the optimal time curvature path obtained in S4.2, calculate the three-dimensional deviation vector between the trainee trajectory and the adaptive standard template in each alignment frame to form a spatiotemporal deviation vector field. Then, using the main motion direction analysis and relative joint reference frame decomposition, decouple the joint trajectory deviation characteristics, center of gravity transfer deviation characteristics, and temporal offset characteristics.

[0030] Preferably, the implementation process of the heterogeneous graph attention error diagnosis network in step S5 is as follows:

[0031] S5.1: Construct a heterogeneous graph attention error diagnosis network, which includes three heterogeneous connection structures: joint space relation graph, motion phase connection graph, and deviation propagation hyperedge;

[0032] S5.2: Map the spatiotemporal deviation vector field and decoupling features output by S4 to the initial features of each node in the graph, and use multi-layer heterogeneous graph attention convolution to pass messages and learn the propagation mode of deviation between different joints and different motion phases.

[0033] S5.3: Connect three parallel classification heads at the end of the network to output the binary classification result of whether each joint is an erroneous joint, the multi-class classification result of the error type, and the error cause label. Based on the three-dimensional deviation vector of the joint in the deviation vector field, generate fine-grained error correction prompts containing the joint name, deviation direction, and offset distance.

[0034] Preferably, the implementation process of the conditional temporal trajectory reconstruction module in step S6 is as follows:

[0035] S6.1: Use the trajectory continuity detector to traverse the physically consistent trainee motion trajectory sequence output by S2. When the trajectory confidence of any joint on a continuous frame is lower than the set threshold or when there is a sudden change in inter-frame displacement that exceeds the limits of human motion, mark the frame segment as a missing segment.

[0036] S6.2: Start the conditional temporal trajectory reconstruction module. This module is a conditional temporal generative adversarial network. The generator adopts a gated recurrent unit structure. It uses the physically consistent joint trajectory of each preset number of frames before and after the missing segment as the temporal context and the spatiotemporal deviation vector field of the corresponding time window in S4 as the conditional input to generate the three-dimensional trajectory coordinates of the missing joint in the missing frame.

[0037] S6.3: Replace the original missing segments with the completed joint trajectories generated in S6.2 to obtain the completed student motion trajectory sequence. Re-input the completed student motion trajectory sequence into S5 to perform error diagnosis.

[0038] This invention also proposes a system for evaluating and correcting dance movement trajectories based on 3D motion capture, the system comprising:

[0039] A multimodal edge collaborative motion capture device, which uses multiple synchronously calibrated depth cameras and wearable inertial measurement units to perform synchronous acquisition of multimodal data and Bayesian multimodal adaptive fusion to generate an initial spatiotemporal trajectory sequence of the trainee's three-dimensional skeleton joints;

[0040] The real-time reconstruction module for physically consistent trajectories uses a spatiotemporal decoupling graph convolutional network and a pre-kinematic constraint network to perform spatiotemporal feature extraction of trainee actions and real-time correction of physical consistency, and outputs a sequence of physically consistent trainee action trajectories.

[0041] An adaptive standard trajectory generation module is used to generate an adaptive rhythm dynamic trajectory template by extracting rhythm density curves and redirecting optimal transmission time, and outputs an adaptive standard trajectory template that is aligned with the rhythm of the trainee's movements.

[0042] A multi-scale trajectory deviation decoupling module is used to perform spatiotemporal deviation vector field calculation and deviation feature decoupling through a phase-sensitive dynamic trajectory warping algorithm and deviation decomposition.

[0043] The heterogeneous graph attention error diagnosis module is used to perform error joint localization, error type classification, error cause attribution, and fine-grained error correction prompt generation through the heterogeneous graph attention error diagnosis network.

[0044] The closed-loop trajectory reconstruction and re-diagnosis module is used to perform missing trajectory completion and secondary error diagnosis after completion through a closed-loop connection between the conditional temporal trajectory reconstruction module and the heterogeneous graph attention error diagnosis module.

[0045] Preferably, the real-time reconstruction module for physically consistent trajectories includes:

[0046] The spatiotemporal decoupling graph convolution submodule is used to extract features from human skeleton joints by performing spatial graph convolution and dilated temporal convolution.

[0047] Gated fusion unit is used to fuse the output features of spatial branches and temporal branches;

[0048] The skeleton length constraint submodule is used to perform scale constraint correction on the trajectory based on the statically calibrated skeleton length of the trainee.

[0049] The joint angle limiting submodule is used to perform physiological limiting projection correction on joint angles based on the human joint range of motion library.

[0050] Preferably, the closed-loop trajectory reconstruction and re-diagnosis module includes:

[0051] The trajectory continuity detection submodule is used to detect the confidence of joint trajectories and inter-frame displacement abrupt changes frame by frame to mark missing segments;

[0052] The Conditional Temporal Generative Adversarial Network submodule is used to generate a complete trajectory based on the physically consistent trajectory in the neighborhood of the missing segment and the corresponding spatiotemporal deviation vector field.

[0053] The closed-loop control submodule is used to automatically input the completed trainee motion trajectory sequence into the heterogeneous graph attention error diagnosis module for secondary diagnosis.

[0054] Compared with the prior art, the beneficial effects of the present invention are:

[0055] This invention utilizes a multimodal edge-cooperative motion capture system combined with Bayesian multimodal adaptive fusion to synchronously acquire data from a depth camera and an inertial measurement unit (IMU) and estimate noise covariance online. This enables stable acquisition of the initial spatiotemporal trajectory sequence of the student's 3D skeleton joints in complex dynamic backgrounds and limb self-occlusion scenarios. Through a spatiotemporally decoupled graph convolutional network and a pre-existing kinematic constraint network, joint features of joint spatial dependence and global temporal dependence are extracted. Skeletal length constraints and physiological limit projection corrections for joint angles are applied to eliminate trajectory distortion and output a student motion trajectory sequence that satisfies human physical consistency constraints. An adaptive rhythm dynamic trajectory template generator extracts the rhythm density curves of the student and teacher and uses a one-dimensional optimal transport mapping to solve the nonlinear time transformation function, redirecting the time axis of the teacher's standard trajectory to eliminate rhythm differences between teacher and student and achieve precise alignment of the motion template and student's movements in motion phase. Finally, a phase-sensitive dynamic trajectory warping algorithm and a multi-scale trajectory deviation decoupling module are used to perform multi-joint joint constraints. The optimal time curvature path is solved, and the deviation vector field is decoupled into joint trajectory deviation features, center of gravity transfer deviation features, and temporal offset features. Different error modes are separated from the unified deviation data, error locations are located, and error causes are distinguished. Through a heterogeneous graph attention error diagnosis network, a heterogeneous graph structure containing joint spatial relationship edges, motion phase connection edges, and deviation propagation hyperedges is constructed to perform attention message passing on multiple types of edges. Three parallel classification heads are used to locate the faulty joints, classify the error types, and infer the error cause attribution, generating fine-grained error correction prompts containing specific joint names, deviation directions, and quantized offset distances. Through the trajectory continuity detection and conditional temporal trajectory reconstruction module, the physical consistency trajectory in the neighborhood of the missing segment and the corresponding spatiotemporal deviation vector field are used as conditions. A conditional temporal generative adversarial network is used to generate a complete trajectory, and the completed trajectory is re-input into the error diagnosis network to form a closed-loop error correction evaluation. Even after severe self-occlusion or dynamic interference causes trajectory loss, the evaluation of the entire sequence can still be completed without interruption.

[0056] In summary, this invention features objective quantitative assessment, real-time feedback, fine-grained error location and cause localization, and robustness against occlusion interference. It is suitable for online or offline auxiliary teaching scenarios in the field of dance education and training, and can effectively reduce the burden on teachers and improve the efficiency of students' movement correction. Attached Figure Description

[0057] Figure 1 This is a flowchart of the dance movement trajectory error correction and evaluation method based on three-dimensional motion capture according to the present invention. Detailed Implementation

[0058] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0059] Please see Figure 1 This invention proposes a dance motion trajectory error correction and evaluation method based on three-dimensional motion capture. The method is implemented based on a multimodal edge collaborative motion capture system and includes the following steps:

[0060] S1: Using multiple synchronously calibrated depth cameras and wearable inertial measurement units, color-depth image sequences and limb inertial motion data of the student's dance movements are collected synchronously. The Bayesian multimodal adaptive fusion algorithm is used to generate the initial spatiotemporal trajectory sequence of the student's three-dimensional skeleton joints.

[0061] S2: Input the initial spatiotemporal trajectory sequence of the student's 3D skeleton joints generated in S1 into the spatiotemporal decoupling graph convolutional network, extract the joint local spatial dependence features and global temporal evolution features of the student's movements, and perform real-time physical consistency correction of joint angles and bone length through the pre-kinematic constraint network, and output a physically consistent student movement trajectory sequence.

[0062] S3: Obtain the standard three-dimensional skeleton trajectory sequence of the teacher's standard dance movements. Using the rhythm density curve extracted from the physically consistent student movement trajectory sequence output by S2 as a condition, use the adaptive rhythm dynamic trajectory template generator to perform non-linear time scaling on the teacher's standard three-dimensional skeleton trajectory sequence to generate an adaptive standard trajectory template that is aligned with the rhythm of the student's movements.

[0063] S4: Input the physically consistent trainee motion trajectory sequence output by S2 and the adaptive standard trajectory template output by S3 into the multi-scale trajectory deviation decoupling module. Use the phase-sensitive dynamic trajectory warping algorithm to calculate the three-dimensional deviation vector of each skeleton joint point in each time phase, form a spatiotemporal deviation vector field, and decouple the joint trajectory deviation characteristics, center of gravity transfer deviation characteristics and temporal offset characteristics from it.

[0064] S5: Construct a heterogeneous graph attention error diagnosis network. The spatiotemporal deviation vector field output by S4 and the decoupled joint trajectory deviation features, center of gravity transfer deviation features and temporal offset features are used as input to this network. The output is the spatial localization of the faulty joint, the error type classification result and the error cause attribution label. Fine-grained error correction prompts containing specific joint names, deviation directions and quantized offset distances are generated.

[0065] S6: Perform joint trajectory continuity detection on the physically consistent trainee motion trajectory sequence output by S2. When a missing frame segment of the trajectory is identified due to self-occlusion or complex dynamic background, the conditional temporal trajectory reconstruction module is activated. Using the physically consistent joint trajectory in the neighborhood of the missing segment in S2 and the spatiotemporal deviation vector field of the corresponding time window in S4 as joint conditions, a complete sequence of missing joint trajectories is generated. The completed trainee motion trajectory sequence is then re-input into S5 for error diagnosis, forming a closed-loop error correction evaluation.

[0066] It should also be noted that the implementation process of the Bayesian multimodal adaptive fusion algorithm in step S1 is as follows:

[0067] S1.1: Perform frame-by-frame human body region segmentation on color-depth image sequences acquired by multiple depth cameras, regress two-dimensional heatmaps of each joint point using a deep convolutional neural network, and reconstruct the three-dimensional joint point positions through multi-view epipolar geometric triangulation.

[0068] S1.2: Perform attitude calculation on the acceleration, angular velocity and geomagnetic data collected by the wearable inertial measurement unit to obtain the spatial orientation of each limb and the relative rotation quaternion of the joints, and calculate the coordinates of the three-dimensional joints based on the forward kinematics of the human kinetic chain.

[0069] S1.3: Using the visual 3D joint coordinates obtained in S1.1 and the inertial 3D joint coordinates obtained in S1.2 as observations, construct a Bayesian fusion filter based on dynamic confidence weighting, estimate the covariance of observation noise for each modality by jointly estimating the visual reprojection error and the inertial sensor residual, and output the fused initial spatiotemporal trajectory sequence of the student's 3D skeleton joints.

[0070] It should also be noted that the implementation process of the spatiotemporal decoupling graph convolutional network and the pre-kinematic constraint network in step S2 is as follows:

[0071] S2.1: Construct a spatiotemporally decoupled graph convolutional network, which consists of a spatial graph convolutional branch and a temporal convolutional branch in parallel. The spatial graph convolutional branch uses the natural connection relationship of the human skeleton as a basis and adds a learnable adaptive adjacency matrix to encode the spatial coordinates of the joints in each frame of the initial spatiotemporal trajectory sequence output by S1. The temporal convolutional branch uses dilated temporal convolution to perform temporal modeling of the spatial features of the joints in multiple consecutive frames. The output features of the two branches are merged through a gating fusion mechanism to obtain the spatiotemporal features of the trainee's actions.

[0072] S2.2: Input the spatiotemporal features of the trainee’s movements into the pre-kinematic constraint network. This network includes a bone length constraint layer and a joint angle limiting layer. The bone length constraint layer corrects trajectory distortion by forcing the length of each bone segment to be consistent with the trainee’s statically calibrated bone length. The joint angle limiting layer projects and corrects joint angles that exceed physiological limits based on the human joint range of motion library.

[0073] S2.3: The pre-kinematic constraint network outputs a physically consistent sequence of student movement trajectories, in which the coordinates of each joint point satisfy the constraints of constant bone length and physiological limits of joint angles.

[0074] It should also be noted that the implementation process of the adaptive rhythm dynamic trajectory template generator in step S3 is as follows:

[0075] S3.1: Extract rhythm features from the physically consistent trainee motion trajectory sequence output by S2, calculate the peak point sequence of the combined velocity of each joint of the trainee, and generate the trainee rhythm density curve through kernel density estimation.

[0076] S3.2: Generate teacher rhythm density curves using the same rhythm feature extraction method for the teacher's standard three-dimensional skeleton trajectory sequence;

[0077] S3.3: Using the student rhythm density curve as the target distribution and the teacher rhythm density curve as the source distribution, the nonlinear monotonic transformation function of the time axis is solved using the one-dimensional optimal transport mapping. This transformation function is then applied to the time axis of the teacher's standard three-dimensional skeleton trajectory sequence to generate an adaptive standard trajectory template after time retargeting.

[0078] It should also be noted that the implementation process of the phase-sensitive dynamic trajectory straightening algorithm in step S4 is as follows:

[0079] S4.1: Using the physically consistent trainee motion trajectory sequence output by S2 and the adaptive standard trajectory template output by S3 as input, construct the trajectory distance matrix for each joint. The distance metric adopts a composite distance function that combines the Euclidean distance of the joint space and the motion phase.

[0080] S4.2: Apply bone length preservation constraints and joint point local rigidity constraints to all relevant nodes, and solve the globally optimal time curvature path through multi-sequence synchronous dynamic time warping to ensure that the time curvature paths of different joint points satisfy the consistency of spatial structure.

[0081] S4.3: Based on the optimal time curvature path obtained in S4.2, calculate the three-dimensional deviation vector between the trainee trajectory and the adaptive standard template in each alignment frame to form a spatiotemporal deviation vector field. Then, using the main motion direction analysis and relative joint reference frame decomposition, decouple the joint trajectory deviation characteristics, center of gravity transfer deviation characteristics, and temporal offset characteristics.

[0082] It should also be noted that the implementation process of the heterogeneous graph attention error diagnosis network in step S5 is as follows:

[0083] S5.1: Construct a heterogeneous graph attention error diagnosis network, which includes three heterogeneous connection structures: joint space relation graph, motion phase connection graph, and deviation propagation hyperedge;

[0084] S5.2: Map the spatiotemporal deviation vector field and decoupling features output by S4 to the initial features of each node in the graph, and use multi-layer heterogeneous graph attention convolution to pass messages and learn the propagation mode of deviation between different joints and different motion phases.

[0085] S5.3: Connect three parallel classification heads at the end of the network to output the binary classification result of whether each joint is an erroneous joint, the multi-class classification result of the error type, and the error cause label. Based on the three-dimensional deviation vector of the joint in the deviation vector field, generate fine-grained error correction prompts containing the joint name, deviation direction, and offset distance.

[0086] It should also be noted that the implementation process of the conditional temporal trajectory reconstruction module in step S6 is as follows:

[0087] S6.1: Use the trajectory continuity detector to traverse the physically consistent trainee motion trajectory sequence output by S2. When the trajectory confidence of any joint on a continuous frame is lower than the set threshold or when there is a sudden change in inter-frame displacement that exceeds the limits of human motion, mark the frame segment as a missing segment.

[0088] S6.2: Start the conditional temporal trajectory reconstruction module. This module is a conditional temporal generative adversarial network. The generator adopts a gated recurrent unit structure. It uses the physically consistent joint trajectory of each preset number of frames before and after the missing segment as the temporal context and the spatiotemporal deviation vector field of the corresponding time window in S4 as the conditional input to generate the three-dimensional trajectory coordinates of the missing joint in the missing frame.

[0089] S6.3: Replace the original missing segments with the completed joint trajectories generated in S6.2 to obtain the completed student motion trajectory sequence. Re-input the completed student motion trajectory sequence into S5 to perform error diagnosis.

[0090] This invention also proposes a system for evaluating and correcting dance movement trajectories based on 3D motion capture, the system comprising:

[0091] The multimodal edge collaborative motion capture device uses multiple synchronously calibrated depth cameras and wearable inertial measurement units to perform synchronous acquisition of multimodal data and Bayesian multimodal adaptive fusion to generate the initial spatiotemporal trajectory sequence of the trainee's 3D skeleton joints.

[0092] The real-time reconstruction module for physically consistent trajectories uses a spatiotemporal decoupling graph convolutional network and a pre-kinematic constraint network to perform spatiotemporal feature extraction of trainee movements and real-time correction of physical consistency, and outputs a sequence of physically consistent trainee movement trajectories.

[0093] The adaptive standard trajectory generation module extracts rhythm density curves and redirects optimal transmission time to generate adaptive rhythm dynamic trajectory templates, outputting adaptive standard trajectory templates that are aligned with the rhythm of the learner's movements.

[0094] The multi-scale trajectory deviation decoupling module uses a phase-sensitive dynamic trajectory warping algorithm and deviation decomposition to perform spatiotemporal deviation vector field calculation and deviation feature decoupling.

[0095] The heterogeneous graph attention error diagnosis module uses a heterogeneous graph attention error diagnosis network to perform error joint localization, error type classification, error cause attribution, and fine-grained error correction prompt generation.

[0096] The closed-loop trajectory reconstruction and re-diagnosis module is used to perform missing trajectory completion and secondary error diagnosis after completion through the closed-loop connection between the conditional temporal trajectory reconstruction module and the heterogeneous graph attention error diagnosis module.

[0097] It should also be noted that the real-time reconstruction module for physically consistent trajectories includes:

[0098] The spatiotemporal decoupling graph convolution submodule is used to extract features from human skeleton joints by performing spatial graph convolution and dilated temporal convolution.

[0099] Gated fusion unit is used to fuse the output features of spatial branches and temporal branches;

[0100] The skeleton length constraint submodule is used to perform scale constraint correction on the trajectory based on the statically calibrated skeleton length of the trainee.

[0101] The joint angle limiting submodule is used to perform physiological limiting projection correction on joint angles based on the human joint range of motion library.

[0102] It should also be noted that the closed-loop trajectory reconstruction and re-diagnosis module includes:

[0103] The trajectory continuity detection submodule is used to detect the confidence of joint trajectories and inter-frame displacement abrupt changes frame by frame to mark missing segments;

[0104] The Conditional Temporal Generative Adversarial Network submodule is used to generate a complete trajectory based on the physically consistent trajectory in the neighborhood of the missing segment and the corresponding spatiotemporal deviation vector field.

[0105] The closed-loop control submodule is used to automatically input the completed trainee motion trajectory sequence into the heterogeneous graph attention error diagnosis module for secondary diagnosis.

[0106] For examples, please refer to Figure 1 This invention provides a method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture. The multimodal edge-cooperative motion capture system used includes:

[0107] The depth camera uses four AzureKinectDK units with a resolution of 1920×1080 and a frame rate of 30fps, and is connected to the edge computing node via gigabit Ethernet.

[0108] The wearable inertial measurement unit uses 17 XsensMTwAwinda nine-axis wireless sensors, which are respectively attached to the dancer's head, thoracic spine, lumbar spine, shoulders, upper arms, forearms, hands, thighs, calves and feet, with a sampling frequency of 200Hz.

[0109] Edge computing nodes are configured with NVIDIA Jetson AGX Orin modules to run all algorithms in real time;

[0110] The teacher's standard movement library pre-stores standard three-dimensional skeleton trajectory sequences recorded by professional dance teachers using the same system;

[0111] Based on the system, the dance movement trajectory error correction and evaluation method based on three-dimensional motion capture of this invention, in practical applications, has the following steps:

[0112] S1 multimodal data synchronous acquisition and Bayesian fusion to generate initial trajectory:

[0113] Simultaneously trigger 4 depth cameras and inertial measurement units to collect color-depth image sequences of the students' dance movements and acceleration, angular velocity and geomagnetic data of each node;

[0114] Visual 3D joint reconstruction uses the YOLOv8-pose network to extract 2D heatmaps and depth values ​​of 17 skeleton joints frame by frame from the color images of each camera;

[0115] Given that the intrinsic parameter matrix of the k-th camera is The external parameter rotation and translation are The two-dimensional coordinates of key point i in the camera are: Depth is ;

[0116] The three-dimensional coordinates of joint point i are solved by minimizing the reprojection error through multi-view triangulation. The optimization objective is to minimize the sum of squared reprojection errors from each viewpoint;

[0117] After solving the problem frame by frame, the visual 3D joint coordinate sequence is obtained. ;

[0118] Inertial 3D joint estimation uses a Madgwick filter to calculate the attitude of each sensor node based on its acceleration and angular velocity, obtaining the quaternion attitude of each limb segment. Then, starting from the pelvic root joint along the human kinetic chain, the coordinate sequence of each joint is calculated step by step through forward kinematics. ;

[0119] Bayesian multimodal adaptive fusion uses the coordinates of key points in both visual and inertial paths as observations and the coordinates of the real skeleton as state variables. It assumes that the noise of both observation paths is zero-mean Gaussian noise, and the covariance matrix is ​​estimated online by visual reprojection error and inertial acceleration residual, respectively.

[0120] The fused joint coordinates are calculated frame by frame. The fusion weights are determined by the inverse matrix of the observation noise covariance of the two modes, so that the mode with less noise automatically receives a higher weight. The initial spatiotemporal trajectory sequence of the trainee's 3D skeleton joints is output. ;

[0121] The core weighting rule of the fusion process is expressed as:

[0122] ;

[0123] S2 spatiotemporal decoupling graph convolution and kinematic constraints generate physically consistent trajectories:

[0124] S1 obtained Input spatiotemporal decoupling graph convolutional network;

[0125] The network consists of parallel spatial graph convolution branches and temporal convolution branches;

[0126] The spatial graph convolution branch uses the natural connectivity of the human skeleton as the basic adjacency matrix and adds a learnable adaptive adjacency matrix. The two-layer graph convolution extracts the spatial dependency features of joints. The temporal convolution branch uses temporal convolution with progressively increasing dilation factors to cover temporal dependencies at different scales in the receptive field.

[0127] The output features of the two branches are weighted and fused through a learnable gating scalar to obtain the spatiotemporal features of the student's actions;

[0128] This spatiotemporal feature is further input into the preceding kinematic constraint network;

[0129] The network contains two constraint layers: the bone length constraint layer calculates the length of each bone segment in the current frame, compares it with the statically calibrated bone length of the trainee, and adjusts the sub-joint coordinates along the bone vector direction until the lengths are consistent; the joint angle limit layer calculates the angle between adjacent bone segments, projects and truncates angles that exceed physiological limits based on the human joint range of motion library, and corrects the joint position in reverse.

[0130] After these two layers of physical constraint correction, the output is a sequence of student motion trajectories that satisfies the physical consistency of bone length and physiological limits of joint angles in each frame. ;

[0131] S3 Adaptive Rhythm Track Template Generation:

[0132] Will The average velocity of each joint is taken to obtain the student's motion energy curve. The local maxima are detected as the rhythm peak. The Gaussian kernel density with a bandwidth of 0.15 seconds is used to estimate and generate the student's rhythm density curve.

[0133] The teacher's rhythm density curve was obtained by applying the same processing to the teacher's standard three-dimensional skeleton trajectory sequence.

[0134] Solving the time redirection function using a one-dimensional optimal transport map;

[0135] The cumulative distribution functions of the rhythm density curves of the students and teachers are calculated separately. A monotonically increasing mapping from the teacher's time axis to the student's time axis is established using the inverse function of the cumulative distribution function. This mapping is then applied to the time axis of the teacher's trajectory. An adaptive standard trajectory template aligned with the rhythm of the student's movements is generated through interpolation and resampling. ;

[0136] S4 Multi-Scale Trajectory Deviation Decoupling:

[0137] Construct a phase-sensitive dynamic trajectory normalization algorithm for... and Perform alignment and deviation decoupling;

[0138] For each key point, calculate the composite distance matrix between the trainee sequence and the template sequence. The distance metric is a weighted composite of spatial Euclidean distance and motion direction difference, with the spatial distance weight set to 0.7.

[0139] To maintain the spatial consistency of the human body structure, synchronization constraints are applied to 17 joints. When solving for the globally optimal time-curved path, a penalty term for the path difference between adjacent joints is added to the total cost. The constraint objective is expressed as:

[0140] ;

[0141] in , It is the set of adjacent edges of the skeleton;

[0142] Solving this optimization problem yields the time-aligned paths for each key.

[0143] Based on the aligned frame correspondence, the three-dimensional deviation vector of each joint at each time step is calculated to form a spatiotemporal deviation vector field.

[0144] Using the analysis of the main motion direction, the component of the deviation vector along the trajectory tangent direction is separated into temporal offset features, the component along the trajectory normal direction is separated into joint trajectory deviation features, and the horizontal displacement of the pelvic node relative to the supporting foot is used as the center of gravity transfer deviation feature.

[0145] S5 Heterogeneous Graph Attention Error Diagnosis and Correction Tips:

[0146] Construct a heterogeneous graph attention error diagnosis network, using time frame instances of each joint as nodes;

[0147] The network defines three heterogeneous connection structures: joint spatial relationship edges are naturally connected by the human skeleton; motion phase connection edges connect joint pairs that are synchronized in motion phase; and deviation propagation hyperedges connect the set of joints in the same frame whose deviation vector directions have an angle of less than 30 degrees, indicating that the deviation may propagate along the motion chain.

[0148] The network uses multi-layer heterogeneous graph attention convolution for message passing. Each relation type learns attention weights and transformation matrices independently. Node features are updated by weighting and aggregating the features of neighbors under each relation and then fusing them with their own features.

[0149] Three parallel classification heads are connected above the top-level node features: the first classification head outputs the binary classification probability of errors for each joint frame; the second classification head outputs the error type, which includes joint trajectory deviation, center of gravity transfer error, temporal misalignment, and compound error; the third classification head outputs the error cause attribution label, which includes insufficient joint mobility, rhythm perception error, improper force control, and insufficient coordination.

[0150] When a joint frame is determined to be erroneous, the deviation direction and offset distance are directly read from the deviation vector of that frame to generate fine-grained error correction prompt information containing the joint name, deviation direction description and quantized offset distance;

[0151] S6 trajectory continuity detection and closed-loop reconstruction diagnosis:

[0152] For the output of S2 Perform frame-by-frame trajectory continuity detection;

[0153] Calculate the ratio of joint displacement between adjacent frames to the average motion amplitude of the trainee, and map it to confidence level using a negative exponential function;

[0154] When the confidence level is below 0.3 and the number of consecutive frames exceeds 5, the segment is marked as a missing trajectory segment caused by self-occlusion or complex background.

[0155] Start the conditional temporal trajectory reconstruction module, the core of which is a conditional temporal generative adversarial network;

[0156] The generator uses a three-layer gated loop unit, inputting the physically consistent joint trajectories of 15 frames before and after the missing segment as the temporal context, and inputting the spatiotemporal deviation vector field of the corresponding time window in S4 as the condition.

[0157] The discriminator uses a three-layer one-dimensional convolutional network;

[0158] The generator learns through adversarial training to generate joint coordinates of missing frames that match the trainee's movement pattern and current deviation trend, and outputs the completed trajectory.

[0159] The missing segments are replaced with the completed joint trajectory to obtain the complete trajectory. The trajectory is then re-entered into S5 to perform error diagnosis, forming a closed-loop error correction and evaluation process. This ensures that the entire sequence of motion evaluation and error localization can still be completed without interruption even when severe occlusion occurs.

[0160] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0161] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture, the method being implemented based on a multimodal edge collaborative motion capture system, characterized in that, Includes the following steps: S1: Using multiple synchronously calibrated depth cameras and wearable inertial measurement units, color-depth image sequences and limb inertial motion data of the student's dance movements are collected synchronously. The Bayesian multimodal adaptive fusion algorithm is used to generate the initial spatiotemporal trajectory sequence of the student's three-dimensional skeleton joints. S2: Input the initial spatiotemporal trajectory sequence of the student's 3D skeleton joints generated in S1 into the spatiotemporal decoupling graph convolutional network, extract the joint local spatial dependence features and global temporal evolution features of the student's movements, and perform real-time physical consistency correction of joint angles and bone length through the pre-kinematic constraint network, and output a physically consistent student movement trajectory sequence. S3: Obtain the standard three-dimensional skeleton trajectory sequence of the teacher's standard dance movements. Using the rhythm density curve extracted from the physically consistent student movement trajectory sequence output by S2 as a condition, use the adaptive rhythm dynamic trajectory template generator to perform non-linear time scaling on the teacher's standard three-dimensional skeleton trajectory sequence to generate an adaptive standard trajectory template that is aligned with the rhythm of the student's movements. S4: Input the physically consistent trainee motion trajectory sequence output by S2 and the adaptive standard trajectory template output by S3 into the multi-scale trajectory deviation decoupling module. Use the phase-sensitive dynamic trajectory warping algorithm to calculate the three-dimensional deviation vector of each skeleton joint point in each time phase, form a spatiotemporal deviation vector field, and decouple the joint trajectory deviation characteristics, center of gravity transfer deviation characteristics and temporal offset characteristics from it. S5: Construct a heterogeneous graph attention error diagnosis network. The spatiotemporal deviation vector field output by S4 and the decoupled joint trajectory deviation features, center of gravity transfer deviation features and temporal offset features are used as input to this network. The output is the spatial localization of the faulty joint, the error type classification result and the error cause attribution label. Fine-grained error correction prompts containing specific joint names, deviation directions and quantized offset distances are generated. S6: Perform joint trajectory continuity detection on the physically consistent trainee motion trajectory sequence output by S2. When a missing frame segment of the trajectory is identified due to self-occlusion or complex dynamic background, the conditional temporal trajectory reconstruction module is activated. Using the physically consistent joint trajectory in the neighborhood of the missing segment in S2 and the spatiotemporal deviation vector field of the corresponding time window in S4 as joint conditions, a complete sequence of missing joint trajectories is generated. The completed trainee motion trajectory sequence is then re-input into S5 for error diagnosis, forming a closed-loop error correction evaluation.

2. The method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture according to claim 1, characterized in that, The implementation process of the Bayesian multimodal adaptive fusion algorithm in step S1 is as follows: S1.1: Perform frame-by-frame human body region segmentation on color-depth image sequences acquired by multiple depth cameras, regress two-dimensional heatmaps of each joint point using a deep convolutional neural network, and reconstruct the three-dimensional joint point positions through multi-view epipolar geometric triangulation. S1.2: Perform attitude calculation on the acceleration, angular velocity and geomagnetic data collected by the wearable inertial measurement unit to obtain the spatial orientation of each limb and the relative rotation quaternion of the joints, and calculate the coordinates of the three-dimensional joints based on the forward kinematics of the human kinetic chain. S1.3: Using the visual 3D joint coordinates obtained in S1.1 and the inertial 3D joint coordinates obtained in S1.2 as observations, construct a Bayesian fusion filter based on dynamic confidence weighting, estimate the covariance of observation noise for each modality by jointly estimating the visual reprojection error and the inertial sensor residual, and output the fused initial spatiotemporal trajectory sequence of the student's 3D skeleton joints.

3. The method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture according to claim 2, characterized in that, The implementation process of the spatiotemporal decoupling graph convolutional network and the pre-kinematic constraint network in step S2 is as follows: S2.1: Construct a spatiotemporally decoupled graph convolutional network, which consists of a spatial graph convolutional branch and a temporal convolutional branch in parallel. The spatial graph convolutional branch uses the natural connection relationship of the human skeleton as a basis and adds a learnable adaptive adjacency matrix to encode the spatial coordinates of the joints in each frame of the initial spatiotemporal trajectory sequence output by S1. The temporal convolutional branch uses dilated temporal convolution to perform temporal modeling of the spatial features of the joints in multiple consecutive frames. The output features of the two branches are merged through a gating fusion mechanism to obtain the spatiotemporal features of the trainee's actions. S2.2: Input the spatiotemporal features of the trainee’s movements into the pre-kinematic constraint network. This network includes a bone length constraint layer and a joint angle limiting layer. The bone length constraint layer corrects trajectory distortion by forcing the length of each bone segment to be consistent with the trainee’s statically calibrated bone length. The joint angle limiting layer projects and corrects joint angles that exceed physiological limits based on the human joint range of motion library. S2.3: The pre-kinematic constraint network outputs a physically consistent sequence of student movement trajectories, in which the coordinates of each joint point satisfy the constraints of constant bone length and physiological limits of joint angles.

4. The method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture according to claim 3, characterized in that, The implementation process of the adaptive rhythm dynamic trajectory template generator in step S3 is as follows: S3.1: Extract rhythm features from the physically consistent trainee motion trajectory sequence output by S2, calculate the peak point sequence of the combined velocity of each joint of the trainee, and generate the trainee rhythm density curve through kernel density estimation. S3.2: Generate teacher rhythm density curves using the same rhythm feature extraction method for the teacher's standard three-dimensional skeleton trajectory sequence; S3.3: Using the student rhythm density curve as the target distribution and the teacher rhythm density curve as the source distribution, the nonlinear monotonic transformation function of the time axis is solved using the one-dimensional optimal transport mapping. This transformation function is then applied to the time axis of the teacher's standard three-dimensional skeleton trajectory sequence to generate an adaptive standard trajectory template after time retargeting.

5. The method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture according to claim 4, characterized in that, The implementation process of the phase-sensitive dynamic trajectory straightening algorithm in step S4 is as follows: S4.1: Using the physically consistent trainee motion trajectory sequence output by S2 and the adaptive standard trajectory template output by S3 as input, construct the trajectory distance matrix for each joint. The distance metric adopts a composite distance function that combines the Euclidean distance of the joint space and the motion phase. S4.2: Apply bone length preservation constraints and joint point local rigidity constraints to all relevant nodes, and solve the globally optimal time curvature path through multi-sequence synchronous dynamic time warping to ensure that the time curvature paths of different joint points satisfy the consistency of spatial structure. S4.3: Based on the optimal time curvature path obtained in S4.2, calculate the three-dimensional deviation vector between the trainee trajectory and the adaptive standard template in each alignment frame to form a spatiotemporal deviation vector field. Then, using the main motion direction analysis and relative joint reference frame decomposition, decouple the joint trajectory deviation characteristics, center of gravity transfer deviation characteristics, and temporal offset characteristics.

6. The method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture according to claim 5, characterized in that, The implementation process of the heterogeneous graph attention error diagnosis network in step S5 is as follows: S5.1: Construct a heterogeneous graph attention error diagnosis network, which includes three heterogeneous connection structures: joint space relation graph, motion phase connection graph, and deviation propagation hyperedge; S5.2: Map the spatiotemporal deviation vector field and decoupling features output by S4 to the initial features of each node in the graph, and use multi-layer heterogeneous graph attention convolution to pass messages and learn the propagation mode of deviation between different joints and different motion phases. S5.3: Connect three parallel classification heads at the end of the network to output the binary classification result of whether each joint is an erroneous joint, the multi-class classification result of the error type, and the error cause label. Based on the three-dimensional deviation vector of the joint in the deviation vector field, generate fine-grained error correction prompts containing the joint name, deviation direction, and offset distance.

7. The method for error correction and evaluation of dance movement trajectories based on three-dimensional motion capture according to claim 6, characterized in that, The implementation process of the conditional temporal trajectory reconstruction module in step S6 is as follows: S6.1: Use the trajectory continuity detector to traverse the physically consistent trainee motion trajectory sequence output by S2. When the trajectory confidence of any joint on a continuous frame is lower than the set threshold or when there is a sudden change in inter-frame displacement that exceeds the limits of human motion, mark the frame segment as a missing segment. S6.2: Start the conditional temporal trajectory reconstruction module. This module is a conditional temporal generative adversarial network. The generator adopts a gated recurrent unit structure. It uses the physically consistent joint trajectory of each preset number of frames before and after the missing segment as the temporal context and the spatiotemporal deviation vector field of the corresponding time window in S4 as the conditional input to generate the three-dimensional trajectory coordinates of the missing joint in the missing frame. S6.3: Replace the original missing segment with the completed joint trajectory generated in S6.2 to obtain the completed student motion trajectory sequence. Re-input the completed student motion trajectory sequence into S5 to perform error diagnosis.

8. A system applied to the dance movement trajectory error correction and evaluation method based on three-dimensional motion capture as described in any one of claims 1-7, characterized in that, The system includes: A multimodal edge collaborative motion capture device, which uses multiple synchronously calibrated depth cameras and wearable inertial measurement units to perform synchronous acquisition of multimodal data and Bayesian multimodal adaptive fusion to generate an initial spatiotemporal trajectory sequence of the trainee's three-dimensional skeleton joints; The real-time reconstruction module for physically consistent trajectories uses a spatiotemporal decoupling graph convolutional network and a pre-kinematic constraint network to perform spatiotemporal feature extraction of trainee actions and real-time correction of physical consistency, and outputs a sequence of physically consistent trainee action trajectories. An adaptive standard trajectory generation module is used to generate an adaptive rhythm dynamic trajectory template by extracting rhythm density curves and redirecting optimal transmission time, and outputs an adaptive standard trajectory template that is aligned with the rhythm of the trainee's movements. A multi-scale trajectory deviation decoupling module is used to perform spatiotemporal deviation vector field calculation and deviation feature decoupling through a phase-sensitive dynamic trajectory warping algorithm and deviation decomposition. The heterogeneous graph attention error diagnosis module is used to perform error joint localization, error type classification, error cause attribution, and fine-grained error correction prompt generation through the heterogeneous graph attention error diagnosis network. The closed-loop trajectory reconstruction and re-diagnosis module is used to perform missing trajectory completion and secondary error diagnosis after completion through a closed-loop connection between the conditional temporal trajectory reconstruction module and the heterogeneous graph attention error diagnosis module.

9. The system according to claim 8, characterized in that, The physically consistent trajectory real-time reconstruction module includes: The spatiotemporal decoupling graph convolution submodule is used to extract features from human skeleton joints by performing spatial graph convolution and dilated temporal convolution. Gated fusion unit is used to fuse the output features of spatial branches and temporal branches; The skeleton length constraint submodule is used to perform scale constraint correction on the trajectory based on the statically calibrated skeleton length of the trainee. The joint angle limiting submodule is used to perform physiological limiting projection correction on joint angles based on the human joint range of motion library.

10. The system according to claim 8, characterized in that, The closed-loop trajectory reconstruction and re-diagnosis module includes: The trajectory continuity detection submodule is used to detect the confidence of joint trajectories and inter-frame displacement abrupt changes frame by frame to mark missing segments; The Conditional Temporal Generative Adversarial Network submodule is used to generate a complete trajectory based on the physically consistent trajectory in the neighborhood of the missing segment and the corresponding spatiotemporal deviation vector field. The closed-loop control submodule is used to automatically input the completed trainee motion trajectory sequence into the heterogeneous graph attention error diagnosis module for secondary diagnosis.

Images