A motion evaluation and digital human feedback method for a rehabilitation training system

Abstract

This invention discloses a method for motion assessment and digital human feedback for rehabilitation training systems, comprising: extracting postures and comparing key point sequences using publicly available standard motion videos and user-uploaded motion videos; using the DTW algorithm to achieve motion temporal alignment, and saving the quantitative results of key point sequence errors and motion accuracy scores for later training strategy adjustments; using a large model to comprehensively analyze keyframe comparison information and quantitative results to generate natural language feedback containing problem localization and improvement suggestions; converting the feedback content into speech and inputting it into a digital human-driven model to generate a voice-synchronized virtual coach video for intuitive interaction; and dynamically adjusting the next round of training strategies based on assessment results and feedback content to form a personalized rehabilitation training plan. This invention integrates digital human interaction and large model technology, providing a reference for improving user experience while reducing rehabilitation training costs.

Description

Technical Field

[0001] This invention relates to the technical fields of digital therapy and artificial intelligence, and in particular to a method for motion assessment and digital human feedback for rehabilitation training systems. Background Technology

[0002] Upper limb dysfunction, a major sequela of neurological diseases and sports injuries, has become a significant health problem affecting quality of life. The existing rehabilitation system faces severe challenges in terms of service capacity, technical level, and service models. Although progress has been made in rehabilitation robotics technology, the adoption rate of hand function rehabilitation robots is less than 15%, and is mostly limited to tertiary hospitals. Primary care institutions still rely primarily on traditional physical therapy, lacking support from advanced technologies such as virtual reality and biofeedback, making it difficult to guarantee the intensity and continuity of training. The few existing remote rehabilitation platforms can only provide basic video guidance, lacking real-time biofeedback and remote assessment capabilities. Therefore, designing a widely adopted model that can analyze the accuracy of patients' rehabilitation movements and provide professional guidance using virtual reality technology is the focus of this research. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a method for motion assessment and digital human feedback for rehabilitation training systems. By integrating cutting-edge digital human interaction and large language model technology, and supplemented by algorithmic innovation, the method analyzes the patient's training movements in real time and generates highly personalized voice and visual feedback. Thus, without direct human intervention, the method provides patients with professional, accurate, and highly accessible home rehabilitation guidance through digital human technology, effectively enhancing the fun and effectiveness of rehabilitation training.

[0004] To achieve the above objectives, the technical solution provided by this invention is: a method for motion assessment and digital human feedback for rehabilitation training systems, comprising the following steps:

[0005] 1) User Personal Training Information Collection: Users upload information about the body parts that need rehabilitation training. The rehabilitation training system automatically specifies training goals, sets training thresholds for corresponding movements, and simultaneously collects user authentication videos, standard movement videos, and user movement videos. The user authentication videos are collected before rehabilitation training begins to facilitate feedback from the digital human-driven model. The standard movement videos use publicly available rehabilitation training videos, and the user movement videos are collected during the rehabilitation training process. The human posture key point sequences of the standard movement videos and the user movement videos are obtained and saved for subsequent posture comparison and adjustment of rehabilitation training strategies.

[0006] 2) Quantitative motion comparison: The human posture key point sequence of the standard motion video and the human posture key point sequence of the user motion video are compared. The quantitative evaluation results of the dynamic time joint key point sequence error degree and motion accuracy score are output, and the judgment result of whether the standard is met is given based on the preset threshold to provide feedback to the user. The quantitative evaluation results are saved for training process recording, phase progress evaluation and adjustment of the next round of rehabilitation training strategy.

[0007] 3) Large model comparison feedback: Extract multiple keyframe images from the standard action video and the user action video respectively, construct a group of "standard keyframe-user keyframe" comparison input, and input the comparison input and the quantitative evaluation results in step 2) into the multimodal large model to generate feedback content including overall conclusions, a list of key issues and actionable correction suggestions.

[0008] 4) Voice Broadcasting and Digital Human Synthesis: The feedback content is processed into a text script after text normalization, then converted into audio signal via text-to-speech conversion. The audio signal is then input into an improved MuseTalk digital human driving model to generate a digital human output result synchronized with the audio signal. The improved MuseTalk digital human driving model includes: First, adaptive adjustment of the mouth region mask based on facial key points and mouth opening / closing state; Second, temporal smoothing of the mouth position, mask boundary, and driving parameters in consecutive frames of the user authentication video to reduce jitter; Third, semantic segmentation of the feedback content generated in step 3) to generate speech and drive corresponding video segments to improve lip-sync, image stability, and naturalness of explanation. The digital human output result consists of a digital human video and a sequence of digital human lip-sync driving parameters, used to broadcast training feedback to the user. After completing this digital human output result, one round of rehabilitation training is completed.

[0009] 5) Personalized rehabilitation training design: By combining the quantitative assessment results output from step 2) and the feedback content from step 3) in this round of rehabilitation training, the goals and thresholds of the next round of training are continuously adjusted to form a more suitable personalized rehabilitation training design for the user.

[0010] Furthermore, in step 2), the method for implementing the action comparison is as follows:

[0011] First, the human pose key point sequences of the collected standard action videos and user action videos are normalized, that is, the sequence of key points of each video is normalized. Frame number Coordinates of key points The processed keypoint coordinates are obtained after translation and scale normalization. :

[0012] ;

[0013] In the formula, For the first A frame reference center point, which is determined by the midpoint of the left and right hip key points or the left and right shoulder key points; The scaling factor is determined by shoulder width and hip width;

[0014] Subsequently, the angle changes of each joint in the human body over continuous time are recorded to construct a joint angle sequence, which is used to characterize the movement change process. Based on this, the DTW method is used to achieve optimal temporal alignment between standard action videos and user action videos by nonlinearly stretching or compressing the time axis, so as to reduce the impact of the difference in the speed of action execution on the results.

[0015] Finally, the multiple error results after alignment are combined for a fusion score, including joint angle error, key point position error and trajectory error, so as to more accurately evaluate the quality of the user's action, give a quantitative evaluation result, and determine whether the user's action is qualified based on a preset threshold.

[0016] Furthermore, in step 3), extract from the standard motion video Zhang keyframes constitute a set Extract from user action videos as well Zhang keyframes constitute a set ,in and These represent the first two images extracted from standard motion videos and user motion videos, respectively. Frame keyframe, and The multimodal large model is input in pairs to generate feedback content F, wherein the feedback content satisfies The keyframes are extracted based on changes in key points of human posture. A video frame is designated as a keyframe when the change in the angle of a target joint between adjacent video frames exceeds a preset threshold, or when the corresponding action enters a preset action time node. Feedback content for the quantitative evaluation results The output follows a structured approach, including overall conclusions and actionable corrective suggestions, to guide users in their next round of rehabilitation training.

[0017] Furthermore, in step 4), the voice broadcasting and digital human synthesis specifically include:

[0018] After performing text normalization on the feedback content described in step 3), the broadcast text script is obtained, and the broadcast text script is divided into semantic sentences. The text clause, the first The text clause is ,in, ;

[0019] The first a text clause The input text-to-speech module outputs the text and the corresponding text. a text clause The corresponding number audio signal ,in, , For the first audio signal Duration;

[0020] An improved MuseTalk digital human-driven model for audio signals and user-authenticated video input, the model includes an audio feature module, a face and mouth preprocessing module, a latent space generation module, and a VAE decoding and video reconstruction module;

[0021] The audio feature module extracts the information most relevant to lip movements from the input audio signal, including pronunciation content, rhythm, pauses, and changes in volume, and compresses it into a series of feature vectors arranged in time, which are then input to the latent space generation module.

[0022] The face and mouth preprocessing module is responsible for face detection, alignment, face segmentation, and mouth region localization to obtain the face region to be modified. It performs the following operations: processing the input user authentication video frame by frame to obtain the... Frame of face images And from the first Frame of face images Extract facial landmarks, and calculate the first... Frame of face images Mouth opening degree According to the degree of mouth opening For the Frame of face images The mouth area mask is adaptively adjusted, and the synthesis area can be dynamically adjusted according to the actual deformation of the mouth to improve the mouth shape matching and the naturalness of the picture. The face and mouth preprocessing module can also perform temporal smoothing processing on the mouth center position, mask boundary parameters and driving parameters in the continuous frames of the user authentication video to reduce inter-frame jitter.

[0023] The latent space generation module generates synchronized lip shapes in the latent space of the VAE based on the latent variables of the occluded lower half of the face, reference face features, and audio features. Then, the UNet model completes the mouth repair and completion in the latent space.

[0024] The VAE decoding and video reconstruction module is used to restore the mouth-related latent variables generated in the latent space into image frames, so that the mouth shape repair results completed in the latent space are remapped into the visualized face image; then it is fused with the identity information, background content and non-mouth areas in the original video, and the frame-level reconstruction is completed in chronological order, and finally the face video with mouth shape changes synchronized with the input speech content is output.

[0025] Furthermore, in step 5), the personalized rehabilitation training design specifically includes:

[0026] Obtain the quantitative assessment results output in step 2) and the feedback content generated in step 3) of this round of rehabilitation training, and adjust the goals and thresholds of the next round of rehabilitation training based on the quantitative assessment results and the feedback content;

[0027] The quantitative evaluation results include a motion accuracy score, sequence error of each joint key point, and a determination of whether the standard has been met; the motion accuracy score is... ,in, The sequence error of each joint key point is: ,in , Let the total number of joints be ; and let the result of determining whether the standard is met be . ,but:

[0028] ;

[0029] Based on the sequence error of each joint key point Calculate the overall error The overall error for:

[0030] ;

[0031] In the formula, The sequence error of each joint key point The corresponding weights, and satisfying:

[0032] ;

[0033] The feedback includes overall conclusions, a list of key issues, and actionable corrective suggestions; the number of key issues is determined based on the list of key issues. ;

[0034] Scoring based on the accuracy of the action Overall error The result of whether or not the standard is met. and the number of key issues mentioned Calculate training status indicators :

[0035] ;

[0036] In the formula, The overall error The normalization result, , , and Let be the weighting coefficients, and satisfy:

[0037] ;

[0038] Let the training objective of this round of rehabilitation training be: The goal of the next round of training is The goal of the next round of training is... Represented as: ,in, Update the step size to achieve the training objective. As the baseline value;

[0039] Let the training threshold for this round of rehabilitation training be... The threshold for the next round of training is Then the threshold for the next round of training Represented as: ,in, Update the step size to train the threshold;

[0040] The goal of the next round of training and the threshold for the next round of training Apply boundary constraints:

[0041] ;

[0042] ;

[0043] In the formula, The maximum value is the training target. To minimize the training objective, The maximum value of the training threshold. This is the minimum training threshold.

[0044] The goal of the next round of training after constraints and the threshold for the next round of training This will serve as a parameter for the next round of rehabilitation training, in order to create a more personalized rehabilitation training design that is better suited to the user.

[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0046] 1. The posture evaluation method of this invention uses DTW and joint angle analysis, which can handle movements with different speeds and rhythms and is more accurate than simple frame comparison methods.

[0047] 2. This invention integrates a large language model to generate natural language feedback, transforming technical evaluation results into coaching-style guidance, which is more approachable and instructive than mechanical numerical feedback.

[0048] 3. The digital human function of this invention generates personalized video instructions through an improved MuseTalk digital human driving model, which, combined with the user's own template videos, is more personalized and targeted than general video demonstrations.

[0049] 4. This invention integrates computer vision, natural language processing, and digital human technology to achieve full automation from motion capture to feedback generation, which is more efficient and standardized than traditional manual evaluation methods. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the overall logic flow of the method of the present invention.

[0051] Figure 2 This is a flowchart of the action comparison and scoring feedback process of the method of the present invention. Detailed Implementation

[0052] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0053] like Figure 1 As shown in the figure, this embodiment discloses a method for motion assessment and digital human feedback for rehabilitation training systems, the specific details of which are as follows:

[0054] 1) Collection of personal training information of users: Users upload information about the body parts that need to be rehabilitated. The rehabilitation training system automatically specifies the training goals, sets the training thresholds for the corresponding movements, and collects user authentication videos, standard movement videos, and user movement videos. The system obtains and saves the human posture key point sequences of the standard movement videos and user movement videos for subsequent posture comparison and adjustment of rehabilitation training strategies.

[0055] Specifically, the system first receives the target rehabilitation training body part information uploaded by the user, and automatically matches the corresponding training target based on a preset hard-coded fixed threshold of rehabilitation medicine knowledge, serving as a reference standard for subsequent training evaluation. In terms of data acquisition, the system simultaneously acquires user authentication videos, standard movement videos, and user movement videos. The user authentication videos are collected before the formal start of rehabilitation training, primarily used to construct the user's individual appearance characteristics, so that the digital human model can generate more personalized feedback results that fit the user's image. After entering the system, the user-uploaded authentication videos undergo unified preprocessing, including resolution standardization, frame rate alignment, and encoding format conversion, ultimately transcoding to standard MP4 format to ensure stable use by the subsequent digital human-driven model. The standard movement videos are sourced from a publicly available rehabilitation training video resource library, and after manual screening or automatic annotation, their movement standardization and medical rationality are ensured, serving as a reference standard for user comparison during training. The user movement videos are collected in real-time by camera equipment and uploaded to the system during rehabilitation training.

[0056] 2) Quantitative comparison of movements: The key point sequence of human posture in the standard movement video and the key point sequence of human posture in the user movement video are compared. The quantitative evaluation results of the dynamic time joint key point sequence error degree and movement accuracy score are output, and the judgment result of whether the standard is met is given based on the preset threshold and fed back to the user. The quantitative evaluation results are saved for training process recording, phase progress evaluation and adjustment of the next round of rehabilitation training strategy.

[0057] like Figure 2 As shown, after the user's action video is fully captured, a refined motion consistency analysis and quantitative evaluation are conducted on the human posture key point sequences extracted from the standard action video and the user's action video. To reduce the impact of individual body size differences, camera distance, and shooting angle on key point coordinates, the original key point data is normalized, including scale normalization based on the human skeletal structure and relative coordinate transformation, thereby improving the comparability between different samples.

[0058] In the temporal dimension, since the speed and rhythm of user actions may differ from standard actions, the DTW algorithm is introduced to perform non-linear temporal alignment between the keypoint sequences of standard actions and user actions. By adaptively stretching or compressing the time axis, DTW can find the optimal matching path between two sequences while maintaining semantic consistency of actions, thereby effectively reducing the impact of differences in action execution speed on the results and achieving robust comparison of action sequences of different lengths and speeds. At the spatial feature level, key motion features of the action are further extracted, and the relative geometric relationships between key joints are modeled. By calculating joint angles and changes in motion trajectories between joints, a more stable action description feature is constructed.

[0059] In terms of quantitative assessment, a comprehensive similarity scoring mechanism was designed, integrating multi-dimensional indicators such as the sequence distance after DTW alignment, spatial errors of key joint points, and joint angle deviations to form a unified movement accuracy score. Based on pre-set training thresholds, the system further judges whether the user's movements meet the standards, and the judgment results are promptly fed back to the user to guide them in adjusting their movement execution methods. Simultaneously, the quantitative assessment results are fully saved to construct the user's training history, supporting subsequent phased rehabilitation effect evaluations and dynamic optimization and adjustment of the next round of personalized training strategies.

[0060] Specifically, the method for implementing the action comparison is as follows:

[0061] First, the human pose key point sequences of the collected standard action videos and user action videos are normalized, that is, the sequence of key points of each video is normalized. Frame number Coordinates of key points The processed keypoint coordinates are obtained after translation and scale normalization. :

[0062] ;

[0063] In the formula, For the first A frame reference center point, which is determined by the midpoint of the left and right hip key points or the left and right shoulder key points; The scale factor is determined by shoulder width and hip width; normalization can minimize the impact of differences in individual height, shooting position, etc.

[0064] Subsequently, the angle changes of each joint in the human body over continuous time are recorded to construct a joint angle sequence, which is used to characterize the movement change process. Based on this, the DTW method is used to achieve optimal temporal alignment between standard action videos and user action videos by nonlinearly stretching or compressing the time axis, so as to reduce the impact of the difference in the speed of action execution on the results.

[0065] Finally, the multiple error results after alignment are combined for a fusion score, including joint angle error, key point position error and trajectory error, so as to more accurately evaluate the quality of the user's action, give a quantitative evaluation result, and determine whether the user's action is qualified based on a preset threshold.

[0066] 3) Large model comparison feedback: Extract multiple keyframe images from the standard action video and the user action video respectively, construct a set of "standard keyframe-user keyframe" comparison input, and input the comparison input and the quantitative evaluation results in step 2) into the multimodal large model to generate feedback content including overall conclusions, a list of key issues and actionable correction suggestions.

[0067] Based on the completed action quantification assessment, a multimodal large model is introduced to understand and generate semantic feedback for user actions. Multiple keyframe images are extracted from both standard action videos and user action videos, either by uniform temporal sampling or based on key action change points. These images are then synchronously matched with the DTW temporal alignment results from the previous stage to construct a comparison input of "standard keyframes - user keyframes." These keyframe pairs are organized in a structured manner and, together with the quantification assessment results obtained in step 2), serve as the input information for the multimodal model.

[0068] At the model invocation level, the system uses the DashScope API to call the Qwen-VL visual language model to perform inference analysis on the aforementioned multimodal inputs. Qwen-VL can simultaneously understand image content and text / numerical information, thereby performing high-level semantic modeling of the differences between user actions and standard actions. In its specific implementation, the system encapsulates keyframe pairs and their corresponding quantitative evaluation results as prompt inputs, guiding the model to output feedback results from multiple dimensions, including action standardization analysis, key problem localization, and improvement suggestion generation.

[0069] Specifically, extracting from standard action videos Zhang keyframes constitute a set Extract from user action videos as well Zhang keyframes constitute a set ,in and These represent the first two images extracted from standard motion videos and user motion videos, respectively. Frame keyframe, and The multimodal large model is input in pairs to generate feedback content F, wherein the feedback content satisfies The keyframes are extracted based on changes in key points of human posture. A video frame is designated as a keyframe when the change in the angle of a target joint between adjacent video frames exceeds a preset threshold, or when the corresponding action enters a preset action time node. Feedback content for the quantitative evaluation results The output follows a structured approach, including overall conclusions and actionable corrective suggestions, to guide users in their next round of rehabilitation training.

[0070] 4) Voice Broadcasting and Digital Human Synthesis: The feedback content is processed into a text script after text normalization, then converted into audio signal via text-to-speech conversion. The audio signal is then input into an improved MuseTalk digital human driving model to generate a digital human output result synchronized with the audio signal. The improved MuseTalk digital human driving model includes: First, adaptive adjustment of the mouth region mask based on facial key points and mouth opening / closing state; Second, temporal smoothing of the mouth position, mask boundary, and driving parameters in consecutive frames of the user authentication video to reduce jitter; Third, semantic segmentation of the feedback content generated in step 3) to generate speech and drive corresponding video segments to improve lip-sync, image stability, and naturalness of explanation. The digital human output result consists of a digital human video and a sequence of digital human lip-sync driving parameters, used to broadcast training feedback to the user. After completing this digital human output result, one round of rehabilitation training is completed.

[0071] The system first performs text normalization on the feedback content generated by the multimodal large model in the previous stage, converting it into a script format suitable for voice broadcasting. This process mainly includes removing redundant symbols, standardizing sentence expression, and adjusting tone to make the text more natural, fluent, and easy for users to understand. Simultaneously, the system performs basic text validation, such as removing leading and trailing whitespace and detecting empty text or abnormal content, to ensure the stability of the subsequent speech synthesis process. After text preprocessing, the system enters the text-to-speech stage, using Microsoft's Edge-TTS service to convert the normalized text script into an audio file. After speech generation, the system further processes the audio format, such as standardizing key audio parameters and using 16-bit PCM encoding, to meet the input requirements of the digital human-driven model.

[0072] The system inputs the standardized voice signal into the improved MuseTalk digital human driving model. This model can generate highly synchronized digital lip-reading and facial expression animations based on the rhythm, intonation, and content of the input voice, thus achieving a "sound-visual consistency" virtual character broadcasting effect. Combined with previously collected user authentication video information, the model can also generate digital humans that more closely resemble the user's image or style, enhancing the realism and immersion of the interaction.

[0073] Specifically, the voice broadcasting and digital human synthesis include:

[0074] After performing text normalization on the feedback content described in step 3), the broadcast text script is obtained, and the broadcast text script is divided into semantic sentences. The text clause, the first The text clause is ,in, ;

[0075] The first a text clause The input text-to-speech module outputs the text and the corresponding text. a text clause The corresponding number audio signal ,in, , For the first audio signal Duration;

[0076] An improved MuseTalk digital human-driven model for audio signals and user-authenticated video input, the model includes an audio feature module, a face and mouth preprocessing module, a latent space generation module, and a VAE decoding and video reconstruction module;

[0077] The audio feature module extracts the information most relevant to lip movements from the input audio signal, including pronunciation content, rhythm, pauses, and changes in volume, and compresses it into a series of feature vectors arranged in time, which are then input to the latent space generation module.

[0078] The face and mouth preprocessing module is responsible for face detection, alignment, face segmentation, and mouth region localization to obtain the face region to be modified. It performs the following operations: processing the input user authentication video frame by frame to obtain the... Frame of face images And from the first Frame of face images Extract facial landmarks, and calculate the first... Frame of face images Mouth opening degree According to the degree of mouth opening For the Frame of face images The mouth area mask is adaptively adjusted, and the synthesis area can be dynamically adjusted according to the actual deformation of the mouth to improve the mouth shape matching and the naturalness of the picture. The face and mouth preprocessing module can also perform temporal smoothing processing on the mouth center position, mask boundary parameters and driving parameters in the continuous frames of the user authentication video to reduce inter-frame jitter.

[0079] The latent space generation module generates synchronized lip shapes in the latent space of the VAE based on the latent variables of the occluded lower half of the face, reference face features, and audio features. Then, the UNet model completes the mouth repair and completion in the latent space.

[0080] The VAE decoding and video reconstruction module is used to restore the mouth-related latent variables generated in the latent space into image frames, so that the mouth shape repair results completed in the latent space are remapped into the visualized face image; then it is fused with the identity information, background content and non-mouth areas in the original video, and the frame-level reconstruction is completed in chronological order, and finally the face video with mouth shape changes synchronized with the input speech content is output.

[0081] 5) Personalized rehabilitation training design: By combining the quantitative assessment results output from step 2) and the feedback content from step 3) in this round of rehabilitation training, the goals and thresholds of the next round of training are continuously adjusted to form a more suitable personalized rehabilitation training design for the user.

[0082] Specifically, the personalized rehabilitation training design includes:

[0083] Obtain the quantitative assessment results output in step 2) and the feedback content generated in step 3) of this round of rehabilitation training, and adjust the goals and thresholds of the next round of rehabilitation training based on the quantitative assessment results and the feedback content;

[0084] The quantitative evaluation results include a motion accuracy score, sequence error of each joint key point, and a determination of whether the standard has been met; the motion accuracy score is... ,in, The sequence error of each joint key point is: ,in , Let the total number of joints be ; and let the result of determining whether the standard is met be . ,but:

[0085] ;

[0086] Based on the sequence error of each joint key point Calculate the overall error The overall error for:

[0087] ;

[0088] In the formula, The sequence error of each joint key point The corresponding weights, and satisfying:

[0089] ;

[0090] The feedback includes overall conclusions, a list of key issues, and actionable corrective suggestions; the number of key issues is determined based on the list of key issues. ;

[0091] Scoring based on the accuracy of the action Overall error The result of whether or not the standard is met. and the number of key issues mentioned Calculate training status indicators :

[0092] ;

[0093] In the formula, The overall error The normalization result, , , and Let be the weighting coefficients, and satisfy:

[0094] ;

[0095] Let the training objective of this round of rehabilitation training be: The goal of the next round of training is The goal of the next round of training is... Represented as: ,in, Update the step size to achieve the training objective. As the baseline value;

[0096] Let the training threshold for this round of rehabilitation training be... The threshold for the next round of training is Then the threshold for the next round of training Represented as: ,in, Update the step size to train the threshold;

[0097] The goal of the next round of training and the threshold for the next round of training Apply boundary constraints:

[0098] ;

[0099] ;

[0100] In the formula, The maximum value is the training target. To minimize the training objective, The maximum value of the training threshold. This is the minimum training threshold.

[0101] The goal of the next round of training after constraints and the threshold for the next round of training This will serve as a parameter for the next round of rehabilitation training, in order to create a more personalized rehabilitation training design that is better suited to the user.

[0102] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for motion assessment and digital human feedback for rehabilitation training systems, characterized in that, Includes the following steps: 1) User Personal Training Information Collection: Users upload information about the body parts that need rehabilitation training. The rehabilitation training system automatically specifies training goals, sets training thresholds for corresponding movements, and simultaneously collects user authentication videos, standard movement videos, and user movement videos. The user authentication videos are collected before rehabilitation training begins to facilitate feedback from the digital human-driven model. The standard movement videos use publicly available rehabilitation training videos, and the user movement videos are collected during the rehabilitation training process. The human posture key point sequences of the standard movement videos and the user movement videos are obtained and saved for subsequent posture comparison and adjustment of rehabilitation training strategies. 2) Quantitative motion comparison: The human posture key point sequence of the standard motion video and the human posture key point sequence of the user motion video are compared. The quantitative evaluation results of the dynamic time joint key point sequence error degree and motion accuracy score are output, and the judgment result of whether the standard is met is given based on the preset threshold to provide feedback to the user. The quantitative evaluation results are saved for training process recording, phase progress evaluation and adjustment of the next round of rehabilitation training strategy. 3) Large model comparison feedback: Extract multiple keyframe images from the standard action video and the user action video respectively, construct a set of "standard keyframe-user keyframe" comparison input, and input the comparison input and the quantitative evaluation results in step 2) into the multimodal large model to generate feedback content including overall conclusions, a list of key issues and actionable correction suggestions. 4) Voice Broadcasting and Digital Human Synthesis: The feedback content is processed into a text script after text normalization, then converted into audio signal via text-to-speech conversion. The audio signal is then input into an improved MuseTalk digital human driving model to generate a digital human output result synchronized with the audio signal. The improved MuseTalk digital human driving model includes: First, adaptive adjustment of the mouth region mask based on facial key points and mouth opening / closing state; Second, temporal smoothing of the mouth position, mask boundary, and driving parameters in consecutive frames of the user authentication video to reduce jitter; Third, semantic segmentation of the feedback content generated in step 3) to generate speech and drive corresponding video segments to improve lip-sync, image stability, and naturalness of explanation. The digital human output result consists of a digital human video and a sequence of digital human lip-sync driving parameters, used to broadcast training feedback to the user. After completing this digital human output result, one round of rehabilitation training is completed. 5) Personalized rehabilitation training design: By combining the quantitative assessment results output from step 2) and the feedback content from step 3) in this round of rehabilitation training, the goals and thresholds of the next round of training are continuously adjusted to form a more suitable personalized rehabilitation training design for the user.

2. The method for motion assessment and digital human feedback for a rehabilitation training system according to claim 1, characterized in that, In step 2), the method for implementing the action comparison is as follows: First, the human pose key point sequences of the collected standard action videos and user action videos are normalized, that is, the sequence of key points of each video is normalized. Frame number Coordinates of key points The processed keypoint coordinates are obtained after translation and scale normalization. : ； In the formula, For the first A frame reference center point, which is determined by the midpoint of the left and right hip key points or the left and right shoulder key points; The scale factor is determined by shoulder width and hip width; Subsequently, the angle changes of each joint in the human body over continuous time are recorded to construct a joint angle sequence, which is used to characterize the movement change process. Based on this, the DTW method is used to achieve optimal temporal alignment between standard action videos and user action videos by nonlinearly stretching or compressing the time axis, so as to reduce the impact of the difference in the speed of action execution on the results. Finally, the multiple error results after alignment are combined for a fusion score, including joint angle error, key point position error and trajectory error, so as to more accurately evaluate the quality of the user's action, give a quantitative evaluation result, and determine whether the user's action is qualified based on a preset threshold.

3. The method for motion assessment and digital human feedback for a rehabilitation training system according to claim 1, characterized in that, In step 3), extract from standard motion video Zhang keyframes constitute a set Extract from user action videos as well Zhang keyframes constitute a set ,in and These represent the first two images extracted from standard motion videos and user motion videos, respectively. Frame keyframe, and The multimodal large model is input in pairs to generate feedback content F, wherein the feedback content satisfies The keyframes are extracted based on changes in key points of human posture. A video frame is designated as a keyframe when the change in the angle of a target joint between adjacent video frames exceeds a preset threshold, or when the corresponding action enters a preset action time node. Feedback content for the quantitative evaluation results The system follows a structured output, including overall conclusions and actionable corrective suggestions, to guide users in their next round of rehabilitation training.

4. The method for motion assessment and digital human feedback for a rehabilitation training system according to claim 1, characterized in that, In step 4), the voice broadcasting and digital human synthesis specifically include: After performing text normalization on the feedback content described in step 3), the broadcast text script is obtained, and the broadcast text script is divided into semantic sentences. The text clause, the first The text clause is ,in, ; The first a text clause The input text-to-speech module outputs the text and the corresponding text. a text clause The corresponding number audio signal ,in, , For the first audio signal Duration; An improved MuseTalk digital human-driven model for audio signals and user-authenticated video input, the model includes an audio feature module, a face and mouth preprocessing module, a latent space generation module, and a VAE decoding and video reconstruction module; The audio feature module extracts the information most relevant to lip movements from the input audio signal, including pronunciation content, rhythm, pauses, and changes in volume, and compresses it into a series of feature vectors arranged in time, which are then input to the latent space generation module. The face and mouth preprocessing module is responsible for face detection, alignment, face segmentation, and mouth region localization to obtain the face region to be modified. It performs the following operations: processing the input user authentication video frame by frame to obtain the... Frame of face images And from the first Frame of face images Extract facial landmarks, and calculate the first... Frame of face images Mouth opening degree According to the degree of mouth opening For the first Frame of face images The mouth area mask is adaptively adjusted, and the synthesis area can be dynamically adjusted according to the actual deformation of the mouth to improve the mouth shape matching and the naturalness of the picture. The face and mouth preprocessing module can also perform temporal smoothing processing on the mouth center position, mask boundary parameters and driving parameters in the continuous frames of the user authentication video to reduce inter-frame jitter. The latent space generation module generates synchronized lip shapes in the latent space of the VAE based on the latent variables of the occluded lower half of the face, reference face features, and audio features. Then, the UNet model completes the mouth repair and completion in the latent space. The VAE decoding and video reconstruction module is used to restore the mouth-related latent variables generated in the latent space into image frames, so that the mouth shape repair results completed in the latent space are remapped into the visualized face image; then it is fused with the identity information, background content and non-mouth areas in the original video, and the frame-level reconstruction is completed in chronological order, and finally the face video with mouth shape changes synchronized with the input speech content is output.

5. The method for motion assessment and digital human feedback for a rehabilitation training system according to claim 1, characterized in that, In step 5), the personalized rehabilitation training design specifically includes: Obtain the quantitative assessment results output in step 2) and the feedback content generated in step 3) of this round of rehabilitation training, and adjust the goals and thresholds of the next round of rehabilitation training based on the quantitative assessment results and the feedback content; The quantitative evaluation results include a motion accuracy score, sequence error of each joint key point, and a determination of whether the standard has been met; the motion accuracy score is... ,in, The sequence error of each joint key point is: ,in , Let the total number of joints be ; and let the result of determining whether the standard is met be . ,but: ； Based on the sequence error of each joint key point Calculate the overall error The overall error for: ； In the formula, The sequence error of each joint key point The corresponding weights, and satisfying: ； The feedback includes overall conclusions, a list of key issues, and actionable corrective suggestions; the number of key issues is determined based on the list of key issues. ; Scoring based on the accuracy of the action Overall error The result of whether or not the standard is met and the number of key issues mentioned Calculate training status indicators : ； In the formula, The overall error The normalization result, , , and Let be the weighting coefficients, and satisfy: ； Let the training objective of this round of rehabilitation training be: The goal of the next round of training is The goal of the next round of training is... Represented as: ,in, Update the step size to achieve the training objective. As the baseline value; Let the training threshold for this round of rehabilitation training be... The threshold for the next round of training is Then the threshold for the next round of training Represented as: ,in, Update the step size to train the threshold; The goal of the next round of training and the threshold for the next round of training Apply boundary constraints: ； ； In the formula, The maximum value is the training target. To minimize the training objective, The maximum value of the training threshold. This is the minimum training threshold. The goal of the next round of training after constraints and the threshold for the next round of training This will serve as a parameter for the next round of rehabilitation training, in order to create a more personalized rehabilitation training design that is better suited to the user.

Images