Real-time posture correction and training method based on multi-modal AI algorithm

Abstract

The application discloses a motion posture real-time correction and training method and system based on a multi-modal AI algorithm, relates to the technical field of artificial intelligence, and comprises the following steps: synchronously collecting multi-modal data streams containing visual data and inertial data to obtain input data in space-time alignment; using a two-stage processing network, generating initial estimation information of human body key points based on the input data, constructing a correlation model between the key points according to the initial estimation information, guiding and optimizing low-confidence key point information through high-confidence key point information, and generating final posture information; comparing with a standard action template, calculating a deviation value, and generating an error diagnosis result; and based on the error diagnosis result, generating a guidance instruction and providing real-time guidance to a user through a preset feedback channel. The application provides an intelligent motion aid that is not constrained by the environment, can understand the deep mechanism of motion, provides instant and personalized feedback, and greatly improves the intelligent and accurate level of motion training.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method for real-time correction and training of motion posture based on a multimodal AI algorithm. Background Technology

[0002] With the rise of the fitness craze and the rapid development of artificial intelligence technology, scientific, personalized, and intelligent sports training has become a research hotspot in the interdisciplinary field of sports science and computer vision. Whether it's professional athletes pursuing ultimate athletic performance, rehabilitation patients undergoing safe recovery training, or ordinary fitness enthusiasts hoping to improve training results and avoid sports injuries, all place extremely high demands on the accuracy and standardization of movement postures. Traditional movement posture correction relies heavily on the visual observation and guidance of experienced coaches or therapists. This method has inherent limitations such as strong subjectivity, difficulty in quantification, limited attention and time, and inability to provide immediate feedback.

[0003] Existing motion posture analysis technologies still face several core challenges in achieving high-precision, high-reliability real-time correction and training. These include: insufficient environmental sensitivity and robustness, significantly reducing the stability of correction systems in real-world scenarios; limited information dimensions and a lack of deep state perception, failing to acquire crucial internal biomechanical information such as muscle activation levels, force distribution, and balance; difficulty in balancing real-time performance and accuracy; and limited feedback dimensions and low personalization, hindering the provision of adaptive, progressive, and personalized training programs based on user fitness levels, fatigue levels, and historical data. Therefore, leveraging the complementary advantages of deep learning models across different modalities to construct an intelligent motion assistance platform that is unconstrained by the environment, capable of understanding deep motion mechanisms, and providing immediate and personalized feedback is a pressing issue that needs to be addressed. Summary of the Invention

[0004] To address the aforementioned technical challenges, this invention proposes a method and system for real-time motion posture correction and training based on a multimodal AI algorithm. This provides intelligent motion assistance that is unconstrained by the environment, capable of understanding the deep mechanisms of motion, and offers immediate and personalized feedback, thereby significantly improving the intelligence and precision of motion training.

[0005] The first aspect of this invention provides a method for real-time motion posture correction and training based on a multimodal AI algorithm, comprising the following steps: S101: Synchronously acquire a multimodal data stream containing visual data and inertial data, and preprocess the multimodal data stream to obtain spatiotemporally aligned input data; S102: Input the input data into a pre-trained two-stage processing network. In the initial estimation stage, based on the input data, generate initial estimation information of human key points. In the relationship optimization stage, based on the initial estimation information, construct a correlation model between key points. Guide and optimize low-confidence key point information through high-confidence key point information to generate optimized final pose information. S103: Compare the final posture information with the standard action template, calculate the deviation value, and generate an error diagnosis result based on the deviation value; S104: Based on the error diagnosis results, generate guidance instructions and provide real-time guidance to the user through a preset feedback channel, while adaptively adjusting training parameters according to the user's historical data.

[0006] In this scheme, step S101, multimodal data synchronous acquisition and preprocessing, includes: The system synchronously acquires RGB video streams and skeletal motion data streams during user movement using a visual sensor and an inertial measurement unit, generating a multimodal data stream containing both visual and inertial data. Based on the timestamp of the visual data, the nearest inertial data sample before and after each video frame is found on the timeline. For the inertial data sample, an interpolation algorithm is used to calculate the inertial data sample that perfectly matches the timestamp of each video frame, and timestamp alignment is performed. The acquired visual and inertial data are denoised and calibrated. Spatial alignment is performed using the transformation matrix between all inertial measurement unit coordinate systems and the camera coordinate system. Finally, the aligned multimodal data is standardized to obtain spatiotemporally aligned input data.

[0007] In this scheme, step S102, during the initial estimation stage, generates initial estimation information for human body key points based on the input data, including: The input data is fed into a two-stage processing network. In the initial estimation stage of the two-stage processing network, the visual data is fed into the visual feature extraction branch built on the HRNet network. The HRNet network outputs a multi-scale visual feature map by connecting sub-networks from high resolution to low resolution in parallel and exchanging information repeatedly between different resolutions. The inertial data is organized into an IMU sensor data sequence, which is then input into the inertial sequence feature extraction branch built based on a Seq2Seq encoder. A gated recurrent unit is used to build an encoder to encode the input IMU sensor data sequence into a context vector that represents the motion dynamics features and time-series dependencies. The context vector is expanded in spatial dimension through a fully connected layer and converted into a feature block that matches the number of channels of the output feature map of the HRNet network. The feature block is then concatenated with the visual feature map to generate a visual-inertial joint feature map. The visual-inertial joint feature map is imported into a Transformer-based keypoint prediction head. The receptive field is increased by introducing dilated convolution and further aggregated. Pointwise convolutional layers are used to generate an initial heatmap for each keypoint. The value of each pixel in the initial heatmap represents the probability that the pixel location is a keypoint. Meanwhile, the prediction head uses a parallel fully connected network to analyze the features of the region corresponding to each key point, outputs an initial confidence score that characterizes the reliability of the prediction, and finally obtains initial estimation information including the initial heatmap, initial confidence score and multimodal feature representation.

[0008] In this scheme, step S102, during the relationship optimization stage, involves constructing a correlation model between key points based on the initial estimation information, including: The initial confidence score of each key point is extracted from the initial estimation information. A confidence threshold is preset. If the initial confidence score is less than the confidence threshold, the corresponding key point is divided into a guidance region. Otherwise, the corresponding key point is divided into a region to be indexed. The guidance region and the region to be indexed are converted into a guidance matrix and a region to be indexed matrix. Based on the initial confidence scores of key points in the guidance region, the guidance matrix is ​​divided into K groups in the channel dimension. Each guidance group is used as an attention head. The query vector is generated by the matrix to be indexed through a learnable linear projection layer, the key vector is generated by the current guidance group matrix through another linear projection layer, and the value vector is generated by the current guidance group matrix through a third linear projection layer. Attention scores are obtained by multiplying the transpose of the query vector and the key vector. The attention scores are then standardized to obtain correlation coefficient matrices from different guidance groups. The correlation coefficient matrices are then combined with the value vector to generate the output of the attention head. Each attention head is assigned a learnable influence factor weight, and the outputs of multiple attention heads are weighted and summed using the influence factor weights to obtain the total correlation matrix.

[0009] In this scheme, in step S102, during the relationship optimization stage, the low-confidence key point information is guided and optimized using high-confidence key point information to generate the optimized final pose information, specifically as follows: In the relationship optimization stage, the guidance matrix contains high-confidence key point information, and the indexing matrix contains low-confidence key point information. The total relevance matrix and the guidance matrix are multiplied to obtain the optimized indexing matrix. The optimized matrix to be indexed and the guiding matrix are reassembled in the original node order to form a node feature matrix. The reassembled node feature matrix is ​​then mapped from the feature space back to the image space through a decoder to generate a corrected heatmap with the same size as the initial heatmap. The generated corrected heatmap is added pixel by pixel to the initial heatmap to generate a high-precision predicted heatmap. Key points are regarded as nodes, and node feature vectors are extracted from the high-precision predicted heatmap. Physical connections are obtained using human skeleton connection relationships, and statistical connections are obtained using key point pairs of common motion learned from the dataset. An edge structure is constructed using the physical connections and statistical connections, and a graph structure is established based on the nodes and the edge structure. An improved graph attention network is used to perform representation learning on the graph structure. Multi-head attention is introduced. For each neighbor node of the target node, attention scores are calculated using node features and edge attributes. The attention scores are standardized to obtain attention weights. The linearly transformed features of the neighbor nodes are weighted and summed using the attention weights. The nodes are updated through an activation function. The outputs of the independent attention heads are concatenated to obtain the updated node features. The updated node features are processed using a multilayer perceptron, and the final key point coordinates are obtained by regression, generating the optimized final pose information.

[0010] In this solution, step S103 involves comparing the final posture information with a standard motion template, calculating the deviation value, and generating an error diagnosis result based on the deviation value, including: An AI evaluation agent for assessing motion posture quality is constructed. In the AI ​​evaluation agent, dynamic time warping is used to align the time axis of the user's action sequence with the time axis of the standard template, and the coordinates of all key points of the user and the template are transformed into a relative coordinate system with the human body core as the far point and the size is normalized. In the AI ​​evaluation agent, in each corresponding frame after time warping, the Euclidean distance between the user and the template is calculated to obtain the spatial coordinate deviation of the key points; the functional joint angle is selected, and the angle difference between the user and the template is calculated in each corresponding frame to obtain the joint angle deviation; the motion trajectory of the preset key parts is obtained, and the Frescher distance between the user trajectory and the template trajectory is calculated to obtain the motion trajectory deviation; in the standard action template, key actions are selected according to the training relevance, and the time proportion of the key action phase is calculated to differ from the template. The variance of the position is calculated to analyze the swaying amplitude of the core parts during the action, and the rhythm and stability deviations are obtained based on the difference and swaying amplitude. Different weights are assigned to the deviations of different dimensions using the currently evaluated action type to obtain a weighted comprehensive deviation score. The weighted comprehensive deviation score is used for error diagnosis. When the weighted comprehensive deviation score is greater than a preset score threshold, it is determined to be an erroneous action. In addition, the AI ​​evaluation agent incorporates a rule base based on exercise physiology, biomechanics, and expert knowledge. It compares various deviation values ​​with preset deviation thresholds and applies logical rules for diagnosis, obtaining the main errors, error details, timing of occurrence, and potential risks, and finally generates error diagnosis results.

[0011] In this solution, step S104 involves generating guidance instructions based on the error diagnosis results and providing real-time guidance to the user through a preset feedback channel, including: The error diagnosis results are imported into a pre-trained large language model in a structured data form, combined with the current user status information and training context. The optimal information is arranged and verbalized according to priority strategy, personalization strategy and instruction generation strategy. The tone and content are adjusted based on user history and historical data, and the deviation value is transformed into an actionable instruction. The guidance instructions are adapted to the visual, auditory, and tactile channels to output augmented reality guidance feedback, voice prompts, and tactile feedback; Continuously track user responses to feedback. If the number of error corrections reaches the preset requirement, gradually reduce the frequency and intensity of feedback. In addition, dynamically adjust the preset score threshold for error judgment based on the user's proficiency.

[0012] In this scheme, step S104 involves adaptively adjusting the training parameters based on historical user data. Collect multi-dimensional data from each user training session, including performance data, fatigue data, and feedback interaction data, and obtain multi-dimensional data within a preset time period as historical data; Based on historical data, a dynamic user model is built, and the average values ​​of accuracy, consistency, stability and completion in the performance data are calculated to form the current capability baseline. Based on the capability baseline, users are classified into predefined levels, and the distribution of deviation data is analyzed to identify users' persistent weaknesses. The training parameters are adaptively adjusted based on the user dynamic matrix, and the user's performance data under the new training parameters is tracked to verify the effectiveness of the adjustment.

[0013] The second aspect of this invention provides a real-time motion posture correction and training system based on a multimodal AI algorithm. The system includes: a multimodal data acquisition and synchronization module, a data preprocessing and spatiotemporal alignment module, a two-stage posture estimation and optimization module, a motion posture evaluation and diagnosis module, a multi-channel feedback generation and scheduling module, and a personalized adaptive management module. The multimodal data acquisition and synchronization module synchronously acquires multimodal data streams including visual data and inertial data, and configures a unified timestamp for all sensor data to ensure that multi-source data are aligned in time. The data preprocessing and spatiotemporal alignment module preprocesses the multimodal data stream to obtain spatiotemporally aligned input data; The dual-stage pose estimation and optimization module constructs a pre-trained dual-stage processing network. In the initial estimation stage, based on the input data, it generates initial estimation information of human key points. In the relationship optimization stage, based on the initial estimation information, it constructs a correlation model between key points. It guides and optimizes low-confidence key point information through high-confidence key point information to generate optimized final pose information. The motion posture assessment and diagnosis module constructs an AI assessment agent for motion posture quality assessment, compares the final posture information with a standard action template, calculates the deviation value, and generates an error diagnosis result based on the deviation value. The multi-channel feedback generation and scheduling module generates guidance instructions based on the error diagnosis results, and adapts them with the visual, auditory, and tactile channels to output augmented reality guidance feedback, voice prompt feedback, and tactile feedback. The personalized adaptive management module establishes a dynamic user model based on the user's historical data, including ability baseline, proficiency level, and weaknesses, and adaptively adjusts the training parameters.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention constructs a perception capability that transcends a single modality by fusing visual and inertial measurement data. Visual information provides macroscopic morphological context, while IMU data provides precise local motion dynamics details unaffected by visual occlusion; the two complement each other. In the relationship optimization stage of the two-stage processing network, a correlation model between key points is automatically established by simulating human biomechanical constraints. This mechanism enables high-confidence key points to intelligently guide and correct low-confidence key points whose predictions are blurred due to occlusion, changes in lighting, or rapid movement. Therefore, when facing complex real-world scenarios, the accuracy and stability of the output results achieve a qualitative leap.

[0015] Secondly, from the hardware-synchronized acquisition of multimodal data and efficient feature extraction and fusion algorithms to lightweight keypoint correction and graph attention inference, all aspects have undergone in-depth optimization in computational efficiency, ensuring extremely low latency throughout the entire process from data input to feedback output. This provides users with seamless and instantaneous action feedback, creating an immersive training experience. Users can immediately adjust their movements based on the system's real-time guidance, forming an efficient learning loop.

[0016] By comparing high-precision pose estimation results with standard action templates in an expert knowledge base, subtle motion deviations are accurately identified, and the error type and root cause are determined. Furthermore, by integrating multiple channels such as augmented reality visual overlay, voice prompts, and haptic feedback, intuitive and highly targeted correction guidance is provided to users. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or examples of the present invention, the accompanying drawings used in the embodiments or examples will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained according to these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a real-time motion posture correction and training method based on a multimodal AI algorithm is shown. Figure 2 A flowchart of the pose estimation and key point relationship optimization method for a two-stage processing network is shown. Figure 3 A flowchart is shown for using an AI-evaluated agent to perform posture evaluation. Figure 4 A block diagram of a motion posture real-time correction and training system based on multimodal AI algorithms is shown. Detailed Implementation

[0019] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0021] Figure 1 A flowchart of a method for real-time motion posture correction and training based on multimodal AI algorithms is shown.

[0022] like Figure 1 As shown, this embodiment provides a method for real-time motion posture correction and training based on a multimodal AI algorithm, including: S101: Synchronously acquire a multimodal data stream containing visual data and inertial data, and preprocess the multimodal data stream to obtain spatiotemporally aligned input data; S102: Input the input data into a pre-trained two-stage processing network. In the initial estimation stage, based on the input data, generate initial estimation information of human key points. In the relationship optimization stage, based on the initial estimation information, construct a correlation model between key points. Guide and optimize low-confidence key point information through high-confidence key point information to generate optimized final pose information. S103: Compare the final posture information with the standard action template, calculate the deviation value, and generate an error diagnosis result based on the deviation value; S104: Based on the error diagnosis results, generate guidance instructions and provide real-time guidance to the user through a preset feedback channel, while adaptively adjusting training parameters according to the user's historical data.

[0023] It should be noted that in step S101, raw data is simultaneously captured from different sensors and processed through a series of steps to eliminate errors caused by hardware and the environment, ultimately generating a clean and uniform input data that is aligned in both time and space. RGB video streams and skeletal motion data streams during user movement are simultaneously acquired using visual sensors (such as RGB cameras and depth cameras) and inertial measurement units (IMUs) attached to key parts of the user's body, generating a multimodal data stream containing both visual and inertial data. The visual sensors continuously capture motion videos of the user at a fixed frame rate. Each video frame provides information on the appearance, contour, and position of the human body in two-dimensional or three-dimensional space for macroscopic posture estimation. Each IMU sensor simultaneously measures the triaxial acceleration, triaxial angular velocity, and triaxial geomagnetic direction of its limb at a higher frequency, providing precise dynamic details of joint movement, such as acceleration, deceleration, and rotation. To achieve precise time alignment, the system employs a hardware synchronization mechanism.

[0024] Using the timestamps of the visual data as a reference, the nearest inertial data samples before and after each video frame are found on the timeline. For these inertial data samples, an interpolation algorithm is used to calculate virtual inertial data samples that perfectly match the timestamps of each video frame, and timestamp alignment is performed. The acquired visual and inertial data are then denoised and calibrated. Spatial alignment is performed using the transformation matrix between all inertial measurement unit coordinate systems and the camera coordinate system. Finally, the aligned multimodal data is standardized to obtain spatiotemporally aligned input data. For example, for a given time point t, its data packet contains: timestamp t, the corresponding image frame, and synchronized and calibrated motion data of all IMUs in the camera coordinate system.

[0025] It should be noted that, as Figure 2As shown, in step S102, during the initial estimation stage, visual and motion features are extracted in parallel from the preprocessed aligned data, and high-precision preliminary keypoint information is generated through a complex prediction head. The input data is fed into a two-stage processing network. In the initial estimation stage of the two-stage processing network, visual data is input into the visual feature extraction branch built based on the HRNet network. The HRNet network connects high-resolution to low-resolution sub-networks in parallel and repeatedly exchanges information between different resolutions, always maintaining high-resolution representation, and outputs a multi-scale visual feature map, which not only contains high spatial precision detail information but also integrates rich semantic context information provided by low-resolution features.

[0026] The inertial data is organized into an IMU sensor data sequence and input into the inertial sequence feature extraction branch built based on a Seq2Seq encoder. A gated recurrent unit is used to build an encoder, which analyzes the motion patterns within the entire time segment and encodes the input IMU sensor data sequence into a context vector that represents the motion dynamics and temporal dependencies.

[0027] The context vector is expanded in spatial dimension through a fully connected layer, transforming it into feature blocks that match the number of channels in the HRNet network's output feature map. These feature blocks are then concatenated with the visual feature map, directly injecting global motion dynamics context information into the local spatial visual features to generate a visual-inertial joint feature map. This visual-inertial joint feature map is then imported into a Transformer-based keypoint prediction head. Dilated convolutions are introduced to increase the receptive field and further aggregate the data, enhancing the understanding of occlusion and complex poses. Pointwise convolutional layers are used to generate an initial heatmap for each keypoint, where the value of each pixel represents the probability that the pixel location is a keypoint. Simultaneously, a parallel fully connected network within the prediction head analyzes the features of the region corresponding to each keypoint, outputting an initial confidence score that characterizes the prediction reliability. This score integrates the sharpness of visual features, the consistency of motion features, and the heatmap response strength of the prediction itself, providing a quantitative assessment of the prediction reliability for that point. Finally, initial estimation information, including the initial heatmap, initial confidence score, and multimodal feature representations, is obtained.

[0028] It should be noted that in step S102, during the relationship optimization stage, the initial confidence score of each key point is extracted from the initial estimation information, and a confidence threshold is preset. If the initial confidence score is less than the confidence threshold, the corresponding key point is divided into a guidance region; otherwise, the corresponding key point is divided into a region to be indexed. The feature representations of high-confidence key points in the guidance region are stacked to form a guidance matrix, and the feature representations of all low-confidence key points transformed in the region to be indexed are stacked to form a matrix to be indexed. Based on the initial confidence scores of key points in the guidance region, the guidance matrix is ​​divided into K groups along the channel dimension, such as high, medium, and low confidence groups. Each guidance group is used as an attention head, and the matrix to be indexed is... Through a learnable linear projection layer Generate query vectors This represents the point to be corrected wanting to know who they should be associated with; determined by the current guidance group matrix. Through another linear projection layer Generate key vectors This represents the identity information that the current group's guidance points can provide; it is derived from the current guidance group matrix. Through the third linear projection layer Generate value vector This represents the valuable information that can be obtained from the current group's guidance point; Attention scores are obtained by multiplying the transpose of the query vector and the key vector, and then standardized to obtain correlation coefficient matrices from different guidance groups. The correlation coefficient matrix and the value vector Combine the output of the generated attention heads; assign learnable influence factor weights to each attention head. The impact factor weight represents the overall importance of the relevance information provided by this set of guide points. The impact factor weight is used... The total relevance matrix is ​​obtained by weighted summation of the outputs of multiple attention heads. .

[0029] In the relation optimization stage, by mining and utilizing the potential statistical and spatial correlations between keypoints, the coarse, potentially noisy, or erroneous results generated in the initial estimation stage are globally optimized and refined. This ultimately outputs a high-precision predicted heatmap after global optimization, generating the final optimized pose information. The total correlation matrix and the guidance matrix are multiplied to obtain the optimized indexing matrix. The new heatmap for each low-confidence keypoint is a weighted sum of the heatmaps for all high-confidence keypoints. For example, a low-confidence point for the left wrist has a high correlation with the left elbow and left shoulder. After optimization, the heatmap for the left wrist will no longer be the initial fuzzy prediction, but will present a reasonable spatial region consistent with the length of the human arm, strongly implied by the heatmaps of the left elbow and left shoulder. The optimized indexing matrix and the guidance matrix are reassembled in the original node order to form a node feature matrix. The reassembled node feature matrix is ​​then mapped back from the feature space to the image space through a decoder, generating a corrected heatmap of the same size as the initial heatmap, reflecting the pose estimation after keypoint relation optimization. The generated corrected heatmap is added pixel-by-pixel to the initial heatmap to generate a high-precision predicted heatmap. The addition operation allows the network to learn only the residual parts that need optimization, rather than completely reconstructing the entire heatmap, making it easier to train and stabilize performance.

[0030] Keypoints are treated as nodes, and node feature vectors are extracted from the high-precision predicted heatmap. Physical connections are obtained using the human skeleton connection relationship, and statistical connections are obtained using keypoint pairs of common motion learned from the dataset. An edge structure is constructed using the physical and statistical connections, and a graph structure is established based on the nodes and the edge structure. An improved graph attention network is used to perform representation learning on the graph structure, introducing multi-head attention. For each neighbor node of the target node, an attention score is calculated using node features and edge attributes (e.g., the limb type or length represented by the edge). The attention scores are standardized to obtain attention weights. The linearly transformed features of neighbor nodes are weighted and summed using the attention weights, and the nodes are updated through an activation function. The outputs of the independent attention heads are concatenated to obtain the updated node features. Operations are performed on the graph through a message passing mechanism. At each layer, each keypoint aggregates information from its connected keypoints. For example, the left wrist node receives information from the left elbow and left hand and updates its own feature representation, explicitly strengthening the biomechanical consistency between keypoints. The updated node features are processed using a multilayer perceptron, and the final key point coordinates are obtained by regression, generating the optimized final pose information.

[0031] It should be noted that, as Figure 3As shown, in step S103, the final posture information is compared with the standard action template. The AI ​​evaluation agent calculates the quantifiable deviation value and ultimately generates a diagnostic report with guiding significance. An AI evaluation agent for assessing motion posture quality is constructed. For continuous movements such as squats and running, the user's movement speed differs from the standard template. In the AI ​​evaluation agent, dynamic time warping is used to non-linearly align the time axis of the user's movement sequence with the time axis of the standard template, ensuring a one-to-one correspondence between the two movement sequences of different durations at key phases such as the squatting, lowest point, and standing up. All key point coordinates of the user and the template are transformed into a relative coordinate system with the human body core as the distal point and their dimensions are normalized.

[0032] In the AI ​​evaluation agent, for each corresponding frame after time warping, the Euclidean distance between the user and the template's corresponding key points is calculated, obtaining a distance curve that changes over time. The agent extracts key statistical features from this curve, such as average deviation, maximum deviation, and the proportion of time exceeding the allowable error threshold, to obtain the spatial coordinate deviation of the key points. Important functional joint angles, such as the knee, hip, and elbow angles, are selected, and the angle difference between the user and the template is calculated for each corresponding frame, obtaining joint angle deviation curves. The agent analyzes these curves to identify insufficient, excessive, or timing errors, thus obtaining joint angle deviations. The motion trajectories of preset key parts such as hands and feet are acquired, and the Fraser distance or Hausdorff distance between the user's trajectory and the template trajectory is calculated. The similarity of the two trajectories in shape and path is comprehensively evaluated, obtaining a scalar value representing the overall trajectory similarity, thus obtaining the motion trajectory deviation. Key movements are selected based on the Pearson correlation coefficient between the training movements and the training objective in the standard movement template. The time proportion of the key movement phase is calculated to differ from the template. By calculating the variance of the position, the amplitude of swaying of core parts such as the torso and pelvis during the movement is analyzed, obtaining rhythm and stability deviations.

[0033] Different movements and training goals have different focuses. For example, strength training focuses more on joint angles and stability. Different weights are assigned to deviations in different dimensions based on the current movement type being evaluated, and a weighted comprehensive deviation score is obtained. The weighted comprehensive deviation score is used for error diagnosis. When the weighted comprehensive deviation score is greater than a preset score threshold, it is judged as an erroneous movement. In addition, a rule base based on exercise physiology, biomechanics and expert knowledge is built into the AI ​​evaluation agent. Various deviation values ​​are compared with preset deviation thresholds, and logical rules are applied for diagnosis to obtain the main errors (the 1-2 most serious and priority problems to be corrected), error details (error type, error location, severity), timing of occurrence (at which stage of the movement the error occurs), and potential risks (indicating the sports injury risk that the error may cause). Finally, an error diagnosis result is generated.

[0034] It should be noted that in step S104, a large language model (LLM) is used to transform the structured diagnostic data into natural and accurate guidance language, and multiple feedback channels are scheduled to output in a coordinated manner to achieve efficient behavior correction.

[0035] Error diagnosis results are imported into a pre-trained large language model in a structured data format, combined with current user status information (training history, fatigue level, etc.) and training context (e.g., the current squat training is the 3rd set). Optimal information arrangement and verbalization are performed based on priority strategies, personalization strategies, and instruction generation strategies. First, major errors in the diagnosis results are addressed to avoid information overload caused by giving too many instructions at once. The tone and content are adjusted based on user history and historical data, using more basic and encouraging language for new users and providing more concise and technical prompts for advanced users. Deviation values ​​are transformed into actionable instructions; for example, a 5-degree pelvic tilt deviation is translated as "Feel your weight evenly distributed on both feet, imagine a string pulling upwards from above your head, and keep your torso upright."

[0036] The guidance instructions are adapted to the visual, auditory, and tactile channels to output augmented reality guidance feedback, voice prompts, and tactile feedback. Augmented reality guidance feedback overlays graphic guidance onto the real-time video feed on the terminal screen, and displays key indicators such as joint angles and depths in real-time on one side of the screen in the form of a dashboard or progress bar, allowing users to intuitively see the gap between themselves and the standard. Voice prompts convert text instructions generated by LLM into voice broadcasts via a TTS engine. Within hundreds of milliseconds after an error occurs, concise and clear instructions are broadcast through headphones or speakers, adjusting the tone and speed according to the severity of the error. Continuous beeping sounds are used to indicate the range of motion, or metronome-like ticking sounds are used to guide the rhythm of the movement. Tactile feedback provides tactile feedback through vibration motors distributed across body parts, providing vibration on the same side of the body where the error occurred. For example, when the left knee buckles inward, a device on the left wrist vibrates; different vibration patterns represent different instructions. Continuous vibration may indicate stopping the current error, while intermittent pulse vibration may indicate paying attention to the rhythm or maintaining the current posture.

[0037] The system continuously tracks user responses to feedback. If the number of error corrections reaches a preset threshold, the frequency and intensity of feedback are gradually reduced, such as downgrading from voice and AR prompts to AR prompts only, to encourage users to develop muscle memory. Additionally, the preset score threshold for error assessment is dynamically adjusted based on the user's proficiency. LLM records common error patterns and progress, providing proactive hints at the start of each training session and generating a personalized summary report after training concludes.

[0038] It should be noted that in step S104, the training parameters are adaptively adjusted based on the user's historical data. Before, during, and after each training session, the user's historical data is analyzed, and a series of training parameters are dynamically adjusted to ensure that the training is always in the optimal challenge range. The adjusted training parameters can both guarantee the training effect and maintain the user's motivation.

[0039] Multi-dimensional data is collected from each user training session, including performance data, fatigue data, and feedback interaction data. Multi-dimensional data within a preset time period is acquired as historical data. Performance data includes accuracy assessment based on the average and maximum deviation values ​​between historical movements and a standard template; consistency assessment based on the variance of deviations between repeated movements within the same set of movements; stability assessment based on the swaying amplitude of core joints such as the pelvis and trunk during movement; and completion assessment based on the number of repetitions and sets completed at the standard amplitude and rhythm. Fatigue data includes morphological decline data and rhythm change data. Morphological decline data indicates whether the accuracy of later repetitions in a set of movements significantly decreases compared to earlier repetitions. Rhythm change data indicates whether the movement speed significantly slows down or loses rhythm due to fatigue. Feedback interaction data includes training frequency based on the user's response speed to guidance instructions and the effectiveness of corrections.

[0040] A dynamic user model is built based on historical data. The average values ​​of accuracy, consistency, stability, and completion in the performance data are calculated to form the current ability baseline. Based on this baseline, users are categorized into predefined levels, such as beginner, intermediate, and proficient. The distribution of deviation data is analyzed to identify persistent weaknesses in users, and targeted corrective exercises are intelligently added to subsequent training. For example, if historical data indicates that a user's core stability is weak, the system may prompt the user to perform a 30-second plank during rest periods between sets, even during bench press training, as an activation exercise. Training parameters are adaptively adjusted based on the dynamic user matrix. If a user consistently exhibits low accuracy or rapid fatigue in a particular movement, the difficulty of the movement is automatically reduced. If the user can easily and effectively complete all sets, the difficulty is increased to provide new stimulation, such as combining two basic movements into a compound movement or prompting an increase in the range of motion. Furthermore, the number of sets, repetitions, and rest time are dynamically adjusted based on user fatigue data. Dynamic deviation score thresholds are set according to different user levels; for example, in beginner mode, a wider deviation score threshold is set, providing alerts only for the most serious and dangerous errors (such as severe knee valgus), avoiding information overload and encouraging beginners to build confidence. User performance data under new training parameters is tracked to verify the effectiveness of adjustments, enabling the system to continuously approach the optimal parameters best suited for the user. By simulating a coach's periodic training mindset, the system automatically plans training cycles.

[0041] Figure 4The architecture diagram of a real-time motion posture correction and training system based on multimodal AI algorithms is shown.

[0042] The second embodiment of the present invention provides a real-time motion posture correction and training system based on multimodal AI algorithm. The system includes: a multimodal data acquisition and synchronization module, a data preprocessing and spatiotemporal alignment module, a two-stage posture estimation and optimization module, a motion posture evaluation and diagnosis module, a multi-channel feedback generation and scheduling module, and a personalized adaptive management module. The multimodal data acquisition and synchronization module synchronously acquires multimodal data streams including visual data and inertial data, and configures a unified timestamp for all sensor data to ensure that multi-source data are aligned in time. The data preprocessing and spatiotemporal alignment module preprocesses the multimodal data stream to obtain spatiotemporally aligned input data; The dual-stage pose estimation and optimization module constructs a pre-trained dual-stage processing network. In the initial estimation stage, based on the input data, it generates initial estimation information of human key points. In the relationship optimization stage, based on the initial estimation information, it constructs a correlation model between key points. It guides and optimizes low-confidence key point information through high-confidence key point information to generate optimized final pose information. The motion posture assessment and diagnosis module constructs an AI assessment agent for motion posture quality assessment, compares the final posture information with a standard action template, calculates the deviation value, and generates an error diagnosis result based on the deviation value. The multi-channel feedback generation and scheduling module uses a large language model to generate guidance instructions based on the error diagnosis results, and adapts them with the visual, auditory, and tactile channels to output augmented reality guidance feedback, voice prompt feedback, and tactile feedback. The personalized adaptive management module establishes a dynamic user model based on the user's historical data, including ability baseline, proficiency level, and weaknesses. It adaptively adjusts training parameters and provides periodic training plan suggestions based on the user's long-term goals and progress.

[0043] The third embodiment of the present invention provides a computer-readable storage medium, which includes a method program for real-time motion posture correction and training based on a multimodal AI algorithm. When the method program is executed by a processor, it implements the steps of the method program for real-time motion posture correction and training based on a multimodal AI algorithm.

[0044] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms. Furthermore, in the various embodiments of the present invention, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0045] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0046] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for real-time motion posture correction and training based on a multimodal AI algorithm, characterized in that, Includes the following steps: S101: Synchronously acquire a multimodal data stream containing visual data and inertial data, and preprocess the multimodal data stream to obtain spatiotemporally aligned input data; S102: Input the input data into a pre-trained two-stage processing network. In the initial estimation stage, based on the input data, generate initial estimation information of human key points. In the relationship optimization stage, based on the initial estimation information, construct a correlation model between key points. Guide and optimize low-confidence key point information through high-confidence key point information to generate optimized final pose information. S103: Compare the final posture information with the standard action template, calculate the deviation value, and generate an error diagnosis result based on the deviation value; S104: Based on the error diagnosis results, generate guidance instructions and provide real-time guidance to the user through a preset feedback channel, while adaptively adjusting training parameters according to the user's historical data.

2. The method for real-time motion posture correction and training based on multimodal AI algorithm according to claim 1, characterized in that, In step S101, the synchronous acquisition and preprocessing of multimodal data includes: The system synchronously acquires RGB video streams and skeletal motion data streams during user movement using a visual sensor and an inertial measurement unit, generating a multimodal data stream containing both visual and inertial data. Based on the timestamp of the visual data, the nearest inertial data sample before and after each video frame is found on the timeline. For the inertial data sample, an interpolation algorithm is used to calculate the inertial data sample that perfectly matches the timestamp of each video frame, and timestamp alignment is performed. The acquired visual and inertial data are denoised and calibrated. Spatial alignment is performed using the transformation matrix between all inertial measurement unit coordinate systems and the camera coordinate system. Finally, the aligned multimodal data is standardized to obtain spatiotemporally aligned input data.

3. The method for real-time motion posture correction and training based on multimodal AI algorithm according to claim 1, characterized in that, In step S102, during the initial estimation stage, based on the input data, initial estimation information for human body key points is generated, including: The input data is fed into a two-stage processing network. In the initial estimation stage of the two-stage processing network, the visual data is fed into the visual feature extraction branch built on the HRNet network. The HRNet network outputs a multi-scale visual feature map by connecting sub-networks from high resolution to low resolution in parallel and exchanging information repeatedly between different resolutions. The inertial data is organized into an IMU sensor data sequence, which is then input into the inertial sequence feature extraction branch built based on a Seq2Seq encoder. A gated recurrent unit is used to build an encoder to encode the input IMU sensor data sequence into a context vector that represents the motion dynamics features and time-series dependencies. The context vector is expanded in spatial dimension through a fully connected layer and converted into a feature block that matches the number of channels of the output feature map of the HRNet network. The feature block is then concatenated with the visual feature map to generate a visual-inertial joint feature map. The visual-inertial joint feature map is imported into a Transformer-based keypoint prediction head. The receptive field is increased by introducing dilated convolution and further aggregated. Pointwise convolutional layers are used to generate an initial heatmap for each keypoint. The value of each pixel in the initial heatmap represents the probability that the pixel location is a keypoint. Meanwhile, the prediction head uses a parallel fully connected network to analyze the features of the region corresponding to each key point, outputs an initial confidence score that characterizes the reliability of the prediction, and finally obtains initial estimation information including the initial heatmap, initial confidence score and multimodal feature representation.

4. The method for real-time motion posture correction and training based on multimodal AI algorithm according to claim 1, characterized in that, In step S102, during the relationship optimization stage, a correlation model between key points is constructed based on the initial estimation information, including: The initial confidence score of each key point is extracted from the initial estimation information. A confidence threshold is preset. If the initial confidence score is less than the confidence threshold, the corresponding key point is divided into a guidance region. Otherwise, the corresponding key point is divided into a region to be indexed. The guidance region and the region to be indexed are converted into a guidance matrix and a region to be indexed matrix. Based on the initial confidence scores of key points in the guidance region, the guidance matrix is ​​divided into K groups in the channel dimension. Each guidance group is used as an attention head. The query vector is generated by the matrix to be indexed through a learnable linear projection layer, the key vector is generated by the current guidance group matrix through another linear projection layer, and the value vector is generated by the current guidance group matrix through a third linear projection layer. Attention scores are obtained by multiplying the transpose of the query vector and the key vector. The attention scores are then standardized to obtain correlation coefficient matrices from different guidance groups. The correlation coefficient matrices are then combined with the value vector to generate the output of the attention head. Each attention head is assigned a learnable influence factor weight, and the outputs of multiple attention heads are weighted and summed using the influence factor weights to obtain the total correlation matrix.

5. The method for real-time motion posture correction and training based on multimodal AI algorithm according to claim 4, characterized in that, In step S102, during the relationship optimization stage, low-confidence keypoint information is guided and optimized using high-confidence keypoint information to generate the optimized final pose information. Specifically: In the relationship optimization stage, the guidance matrix contains high-confidence key point information, and the indexing matrix contains low-confidence key point information. The total relevance matrix and the guidance matrix are multiplied to obtain the optimized indexing matrix. The optimized matrix to be indexed and the guiding matrix are reassembled in the original node order to form a node feature matrix. The reassembled node feature matrix is ​​then mapped from the feature space back to the image space through a decoder to generate a corrected heatmap with the same size as the initial heatmap. The generated corrected heatmap is added pixel by pixel to the initial heatmap to generate a high-precision predicted heatmap. Key points are regarded as nodes, and node feature vectors are extracted from the high-precision predicted heatmap. Physical connections are obtained using human skeleton connection relationships, and statistical connections are obtained using key point pairs of common motion learned from the dataset. An edge structure is constructed using the physical connections and statistical connections, and a graph structure is established based on the nodes and the edge structure. An improved graph attention network is used to perform representation learning on the graph structure. Multi-head attention is introduced. For each neighbor node of the target node, attention scores are calculated using node features and edge attributes. The attention scores are standardized to obtain attention weights. The linearly transformed features of the neighbor nodes are weighted and summed using the attention weights. The nodes are updated through an activation function. The outputs of the independent attention heads are concatenated to obtain the updated node features. The updated node features are processed using a multilayer perceptron, and the final key point coordinates are obtained by regression, generating the optimized final pose information.

6. The method for real-time motion posture correction and training based on multimodal AI algorithm according to claim 1, characterized in that, In step S103, the final posture information is compared with the standard motion template, the deviation value is calculated, and an error diagnosis result is generated based on the deviation value, including: An AI evaluation agent for assessing motion posture quality is constructed. In the AI ​​evaluation agent, dynamic time warping is used to align the time axis of the user's action sequence with the time axis of the standard template, and the coordinates of all key points of the user and the template are transformed into a relative coordinate system with the human body core as the far point and the size is normalized. In the AI ​​evaluation agent, in each corresponding frame after time warping, the Euclidean distance between the user and the template is calculated to obtain the spatial coordinate deviation of the key points; the functional joint angle is selected, and the angle difference between the user and the template is calculated in each corresponding frame to obtain the joint angle deviation; the motion trajectory of the preset key parts is obtained, and the Frescher distance between the user trajectory and the template trajectory is calculated to obtain the motion trajectory deviation; in the standard action template, key actions are selected according to the training relevance, and the time proportion of the key action phase is calculated to differ from the template. The variance of the position is calculated to analyze the swaying amplitude of the core parts during the action, and the rhythm and stability deviations are obtained based on the difference and swaying amplitude. Different weights are assigned to the deviations of different dimensions using the currently evaluated action type to obtain a weighted comprehensive deviation score. The weighted comprehensive deviation score is used for error diagnosis. When the weighted comprehensive deviation score is greater than a preset score threshold, it is determined to be an erroneous action. In addition, the AI ​​evaluation agent incorporates a rule base based on exercise physiology, biomechanics, and expert knowledge. It compares various deviation values ​​with preset deviation thresholds and applies logical rules for diagnosis, obtaining the main errors, error details, timing of occurrence, and potential risks, and finally generates error diagnosis results.

7. The method for real-time motion posture correction and training based on multimodal AI algorithm according to claim 1, characterized in that, In step S104, based on the error diagnosis results, guidance instructions are generated and provided to the user in real time through a preset feedback channel, including: The error diagnosis results are imported into a pre-trained large language model in a structured data form, combined with the current user status information and training context. The optimal information is arranged and verbalized according to priority strategy, personalization strategy and instruction generation strategy. The tone and content are adjusted based on user history and historical data, and the deviation value is transformed into an actionable instruction. The guidance instructions are adapted to the visual, auditory, and tactile channels to output augmented reality guidance feedback, voice prompts, and tactile feedback; Continuously track user responses to feedback. If the number of error corrections reaches the preset requirement, gradually reduce the frequency and intensity of feedback. In addition, dynamically adjust the preset score threshold for error judgment based on the user's proficiency.

8. The method for real-time motion posture correction and training based on multimodal AI algorithm according to claim 1, characterized in that, In step S104, the training parameters are adaptively adjusted based on the user's historical data. Collect multi-dimensional data from each user training session, including performance data, fatigue data, and feedback interaction data, and obtain multi-dimensional data within a preset time period as historical data; Based on historical data, a dynamic user model is built, and the average values ​​of accuracy, consistency, stability and completion in the performance data are calculated to form the current capability baseline. Based on the capability baseline, users are classified into predefined levels, and the distribution of deviation data is analyzed to identify users' persistent weaknesses. The training parameters are adaptively adjusted based on the user dynamic matrix, and the user's performance data under the new training parameters is tracked to verify the effectiveness of the adjustment.

9. A real-time motion posture correction and training system based on multimodal AI algorithms, characterized in that, To implement the real-time motion posture correction and training method based on multimodal AI algorithm as described in any one of claims 1-8, the system includes: a multimodal data acquisition and synchronization module, a data preprocessing and spatiotemporal alignment module, a two-stage posture estimation and optimization module, a motion posture evaluation and diagnosis module, a multi-channel feedback generation and scheduling module, and a personalized adaptive management module. The multimodal data acquisition and synchronization module synchronously acquires multimodal data streams including visual data and inertial data, and configures a unified timestamp for all sensor data to ensure that multi-source data are aligned in time. The data preprocessing and spatiotemporal alignment module preprocesses the multimodal data stream to obtain spatiotemporally aligned input data; The dual-stage pose estimation and optimization module constructs a pre-trained dual-stage processing network. In the initial estimation stage, based on the input data, it generates initial estimation information of human key points. In the relationship optimization stage, based on the initial estimation information, it constructs a correlation model between key points. It guides and optimizes low-confidence key point information through high-confidence key point information to generate optimized final pose information. The motion posture assessment and diagnosis module constructs an AI assessment agent for motion posture quality assessment, compares the final posture information with a standard action template, calculates the deviation value, and generates an error diagnosis result based on the deviation value. The multi-channel feedback generation and scheduling module generates guidance instructions based on the error diagnosis results, and adapts them with the visual, auditory, and tactile channels to output augmented reality guidance feedback, voice prompt feedback, and tactile feedback. The personalized adaptive management module establishes a dynamic user model based on the user's historical data, including ability baseline, proficiency level, and weaknesses, and adaptively adjusts the training parameters.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a method program for real-time motion posture correction and training based on a multimodal AI algorithm. When the method program is executed by a processor, it implements the steps of the method program for real-time motion posture correction and training based on a multimodal AI algorithm as described in any one of claims 1 to 8.

Images