A voice and hand movement synchronous mirror training system

Abstract

The application relates to a voice and hand movement synchronous mirror image training system. The system comprises an information collection layer for capturing real-time movement images and voice data of a patient during training in real time in response to a training start instruction; the real-time movement images comprise real-time movement images of a healthy upper limb and real-time movement images of a diseased upper limb; a mirror image generation unit is used for mirror flipping of the real-time movement images of the healthy upper limb, generating virtual upper limb movement feedback images, and sending the virtual upper limb movement feedback images to a patient display end; an evaluation layer is used for analyzing matching degrees of the real-time movement images and corresponding standard upper limb action demonstration animations and the voice data and corresponding standard voice prompts, and generating a comprehensive rehabilitation progress report. The system can realize synchronous fusion of language rehabilitation and upper limb movement training, objective quantification and individualized regulation of a rehabilitation process, and is suitable for a comprehensive rehabilitation intervention scene of post-stroke multiple functional disorder patients.

Description

Technical Field

[0001] This invention belongs to the field of rehabilitation training technology, and in particular relates to a voice and hand movement synchronized mirror training system. Background Technology

[0002] With the development of rehabilitation medicine and human-computer interaction technology, human-computer interaction rehabilitation training technology based on mirror therapy (MT) has emerged. By presenting the patient's healthy upper limb in a mirror, it creates a visual illusion of normal movement of the affected limb, thereby activating brain regions related to motor control and promoting functional recovery.

[0003] In traditional techniques, post-stroke patients mainly focus on the recovery of motor function, which cannot affect cognitive and language functions, or they undergo motor function recovery training and language function recovery training separately.

[0004] However, the recovery of motor and language functions has a mutually reinforcing effect. Patients receiving language and physical training at different times result in fragmented intervention tasks and disjointed processes, making it difficult to motivate sustained patient participation and hindering the functional network linkages between brain regions. Language training alone also lacks the activation of the motor control system, insufficient to generate broad neuroplasticity changes. Furthermore, the success of both motor and language training largely depends on therapist assessment, lacking precise data-driven motion capture and speech analysis, leading to highly subjective evaluation of training effectiveness and a lack of standardization and continuity. Summary of the Invention

[0005] Therefore, it is necessary to provide a synchronized mirror training system for speech and hand movements that can combine language training and motor training and perform standardized assessments to address the aforementioned technical problems.

[0006] In a first aspect, this application provides a voice and hand movement synchronized mirror training system, comprising:

[0007] The system consists of an information acquisition layer, a processing layer, and an evaluation layer; the processing layer includes a mirror generation unit.

[0008] The information acquisition layer is used to respond to the training start command and capture real-time motion images and voice data of the patient during training. The training start command is used to instruct the patient's display to execute the acquired demonstration content. The demonstration content includes standard voice prompts for playback, as well as standard voice lip-sync animation and standard upper limb movement demonstration animation for display. The real-time motion images include real-time motion images of the unaffected upper limb and the affected upper limb.

[0009] The mirror generation unit is used to mirror and flip the real-time motion image of the unaffected upper limb to generate a virtual upper limb motion feedback image, and then send the virtual upper limb motion feedback image to the patient's display terminal.

[0010] The assessment layer is used to analyze the matching degree between real-time motion images and corresponding standard upper limb movement demonstration animations, as well as voice data and corresponding standard voice prompts, and to generate a comprehensive rehabilitation progress report.

[0011] In one embodiment, the processing layer further includes an instruction decoder and a voice synchronization controller;

[0012] The instruction decoder is used to parse the preset training task configuration file to obtain the corresponding standard voice prompt data, standard voice lip-sync animation data, and standard upper limb movement demonstration animation data.

[0013] The voice synchronization controller is used to align standard voice prompt data, standard voice lip-sync animation data, and standard upper limb movement demonstration animation data along the timeline to obtain the demonstration content.

[0014] In one embodiment, the system further includes an interaction layer; the interaction layer includes a display module and a configuration module;

[0015] The display module is used to instruct the patient's display terminal to alternately display the demonstration content and the virtual upper limb movement feedback image at preset time intervals;

[0016] The configuration module is used to respond to the obtained training action type, number of training actions, and training action rhythm, call the corresponding training action and voice joint template, and generate a training task configuration file.

[0017] In one embodiment, the evaluation layer includes a motion evaluation module, a language evaluation module, and a feedback module;

[0018] The motion assessment module is used to analyze the consistency of motion amplitude between real-time motion images and corresponding standard upper limb motion demonstration animations to obtain the motion completion rate;

[0019] The language evaluation module is used to analyze the consistency between the voiceprint of the speech data and the corresponding standard speech prompts to obtain the accuracy of the language imitation.

[0020] The feedback module is used to generate a comprehensive rehabilitation progress report based on the degree of action completion and the accuracy of language imitation.

[0021] In one embodiment, the consistency of the motion amplitude between real-time motion images and corresponding standard upper limb movement demonstration animations is analyzed to obtain the motion completion rate, including:

[0022] Using a pose estimation algorithm based on human keypoint detection, skeletal point sequences were extracted from real-time motion images of the healthy upper limb and the affected upper limb, resulting in skeletal point sequences of the healthy upper limb and the affected upper limb.

[0023] The motion trajectory of the skeletal point sequence of the healthy upper limb is compared with the standard skeletal point motion trajectory corresponding to the standard upper limb motion demonstration animation to obtain the motion execution correction gap.

[0024] The trajectory of the skeletal point sequence of the healthy upper limb and the skeletal point sequence of the affected upper limb are compared to obtain the difference in action execution.

[0025] The action completion rate is obtained by adjusting the action execution gap using an action execution correction factor.

[0026] In one embodiment, trajectory comparison corresponds to the following steps:

[0027] The difference between the skeletal point sequences is evaluated based on the metrics of angle deviation, path similarity, and velocity curve matching to obtain deviation data;

[0028] The difference in the skeleton point sequence is obtained by calculating the deviation data using the cosine similarity of the pose vectors.

[0029] In one embodiment, analyzing the consistency between the voiceprint of the speech data and the corresponding standard speech prompt to obtain the accuracy of language imitation includes:

[0030] Mel frequency cepstral coefficients are extracted from the speech data to obtain speech features;

[0031] Based on dynamic time warping, speech features are matched with the audio features corresponding to standard speech prompts to obtain the accuracy of speech imitation.

[0032] Secondly, this application also provides a method for synchronized mirror training of speech and hand movements, including:

[0033] In response to the training start command, the demonstration content is sent to the patient's display terminal;

[0034] Real-time motion images and voice data are captured when the patient performs the demonstration content; real-time motion images include real-time motion images of the unaffected upper limb and the affected upper limb.

[0035] The real-time motion image of the unaffected upper limb is mirrored and flipped to generate a virtual upper limb motion feedback image, which is then sent to the patient's display device.

[0036] By analyzing real-time motion images and corresponding standard upper limb movement demonstration animations, as well as the matching degree between voice data and corresponding standard voice prompts, a comprehensive rehabilitation progress report is generated.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described voice and hand movement synchronous mirror training method.

[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described voice and hand movement synchronous mirror training method.

[0039] The aforementioned synchronized mirror training system for speech and hand movements integrates language and movement into a single training process for simultaneous acquisition and processing, overcoming the limitations of traditional rehabilitation training that separates language and movement processing. Through the mirror generation unit in the processing layer, real-time motion images of the unaffected upper limb are flipped to generate fictitious feedback on the affected side, providing a continuous and natural visual illusion of movement on the affected side. This strengthens the brain's recognition and activation of the affected limb, fully utilizing the principle of neuroplasticity and representing an effective extension of traditional mirror therapy on a digital platform. The system guides patients to complete imitation tasks by watching standard speech prompts, standard lip movements, and standard movement demonstrations. This audiovisual mechanism lowers the cognitive threshold for patients to understand and execute training tasks, enhancing their active cooperation and operational accuracy, making it particularly suitable for stroke patients with partial cognitive impairment. The evaluation layer compares the patient's movement trajectory and speech data during execution with standard demonstration data, generating a matching degree analysis result, thus forming a continuous and quantitative rehabilitation progress report. This mechanism solves the problems of strong subjectivity, delayed feedback, and difficulty in tracking training effects in traditional rehabilitation training. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a diagram illustrating the application environment of the voice and hand movement synchronized mirror training system of the present invention.

[0042] Figure 2 This is a structural diagram of the voice and hand movement synchronized mirror training system of the present invention;

[0043] Figure 3 This is a structural diagram of the processing layer of the voice and hand movement synchronous mirror training system of the present invention;

[0044] Figure 4 This is a flowchart illustrating the synchronous mirror training method for voice and hand movements according to the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] The voice and hand movement synchronized mirror training system provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be integrated onto server 102 or placed on a cloud or other network server. Terminal 101 can be, but is not limited to, various personal computers, laptops, and tablets. Terminal 101 can be a single terminal connected to camera 103, allowing both the patient and trainer to complete training task configuration and training on a single terminal. Alternatively, one terminal 101 can be connected to two displays for separate use by the patient and trainer. Two terminals 101 can also be configured, with the patient's terminal connected to camera 103. Camera 103 is used to capture the patient's upper limb movements. An opaque panel or the terminal 101's display screen is used to obscure the patient's true visual range of the limbs, allowing the patient to perceive limb movements only through the terminal 101's display screen. Server 102 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0047] In one exemplary embodiment, such as Figure 2 As shown, a voice and hand movement synchronized mirror training system is provided, including: an information acquisition layer 210, a processing layer 220 and an evaluation layer 230; the processing layer 220 includes a mirror generation unit.

[0048] The information acquisition layer 210 is used to capture real-time motion images and voice data of the patient during training in response to the training start command; the training start command is used to instruct the patient's display terminal to execute the acquired demonstration content; the demonstration content includes standard voice prompts for playback, and standard voice lip-sync animation and standard upper limb movement demonstration animation for display; the real-time motion images include real-time motion images of the unaffected upper limb and the affected upper limb.

[0049] Indicatively, the training start command can be issued by the therapist-side control system or automatically scheduled by local control logic based on the patient's schedule. This command serves as a trigger signal for task scheduling, guiding subsequent modules to perform integrated control over the loading, demonstration, and acquisition of training content. Specifically, it instructs the processing layer to load the demonstration content corresponding to the current training task and instructs the patient's display terminal to execute the display of this content. The demonstration content includes pre-recorded or synthesized standard voice prompt files, standard voice lip-sync animations, and standard upper limb movement demonstration animations. The standard voice prompt files are used to play the standard voice prompts, while the standard voice lip-sync animations and standard upper limb movement demonstration animations are used for display. Optionally, the demonstration content can be sourced from a local resource library or dynamically retrieved from a cloud-based training task database.

[0050] Furthermore, after the demonstration content is ready, the patient's display terminal begins to synchronously play audio and display lip movements and upper limb actions, guiding the patient to understand the current training task and respond in a combined audiovisual manner. The information acquisition layer includes an image acquisition unit composed of a top-down camera or a depth camera, and a voice acquisition unit constructed from a microphone array. Its task is to simultaneously acquire the patient's voice data and motion image data during training execution.

[0051] Optionally, to support limb mirror training, the image acquisition unit can acquire image streams of the healthy and affected upper limbs separately. A dual-camera structure or a wide-angle camera combined with image partitioning can be used to identify the image regions of the upper limbs on different sides through a limb recognition algorithm, thereby extracting two independent motion images of the healthy and affected sides.

[0052] The mirror generation unit is used to mirror and flip the real-time motion image of the unaffected upper limb to generate a virtual upper limb motion feedback image, and then send the virtual upper limb motion feedback image to the patient's display terminal.

[0053] Indicatively, after acquiring the motion image of the unaffected upper limb, the mirror generation unit enters the working state. In order to ensure the real-time display, the mirror generation unit simply flips the real-time motion image of the unaffected upper limb left and right and transmits it to the patient's display terminal. The patient can visually perceive the illusion that their affected limb is performing the correct movement, thereby inducing the activation of the motor cortex and the shaping of the compensatory neural network.

[0054] The assessment layer 230 is used to analyze the real-time motion images and the corresponding standard upper limb movement demonstration animations, as well as the matching degree between voice data and the corresponding standard voice prompts, and to generate a comprehensive rehabilitation progress report.

[0055] In a demonstrative manner, the system simultaneously captures the patient's speech imitation responses in real time while playing standard voice prompts. Specifically, the speech data undergoes denoising and segmentation in a preprocessing module, and then its main speech feature parameters are extracted using Mel-frequency cepstral coefficients (MFCC). Furthermore, the MFCC features of the patient's speech are matched with the corresponding features of the standard voice prompts using dynamic time warping (DTW) analysis to assess their similarity in dimensions such as speech rate, intonation, and articulation, and a speech matching score is generated.

[0056] Simultaneously, the system can also compare and analyze real-time motion images of the affected upper limb with standard movement demonstrations. Specifically, it compares the consistency between the movement trajectory of the healthy side and the standard demonstration movement to verify the accuracy of the healthy side's imitation. Furthermore, it compares the deviation between the movement of the affected side and the mirrored movement of the healthy side, i.e., whether the affected side has followed visual guidance to a certain extent and completed a similar movement trajectory. Optionally, the comparison method can be based on skeletal data to calculate posture similarity, or it can combine dynamic features such as movement speed and acceleration curves, and score through dynamic curve fitting or neural network models.

[0057] The assessment layer integrates the results of voice and movement comparisons and generates a comprehensive rehabilitation progress report for the current training round according to preset scoring weights. Optionally, the system can also archive data from multiple training rounds in a time series to construct an individualized rehabilitation progress curve and provide graphical representation. If the matching degree continuously declines or synchronization deviations occur during the training process, the system will issue a prompt, suggesting that the therapist adjust the task pace or content.

[0058] In the aforementioned synchronized mirror training system for voice and hand movements, the information acquisition layer can simultaneously collect the patient's voice data and upper limb movement image data during training. Compared to traditional training systems that only collect single movement or voice information, this significantly improves the completeness and perceptual dimension of the training interaction. The mirror generation unit mirrors the real-time motion image of the unaffected upper limb, generating a visually symmetrical virtual upper limb motion feedback image. This feedback image is displayed in real-time on the patient's screen, creating the visual illusion that the affected limb is moving normally. This promotes the reactivation of the affected limb in the brain's motor cortex, strengthens neural plasticity mechanisms, and helps improve upper limb function recovery in hemiplegic patients. The training start command triggers the playback of demonstration content on the patient's terminal. This demonstration content includes standard voice prompts, standard voice lip-sync animations, and standard upper limb movement demonstration animations. This allows for simultaneous guidance of the training task in three dimensions: voice prompts, movement observation, and imitation. This improves the patient's understanding and willingness to execute the training task, and is beneficial for increasing active participation and compliance in rehabilitation training. By analyzing the matching degree between the movement and voice data collected during training and preset standards through an assessment layer, a comprehensive rehabilitation progress report is generated after training is completed. This design enables real-time assessment and stage feedback during the training process, avoiding the problems of unquantifiable training processes and untrackable effects in traditional rehabilitation methods. It provides therapists with objective and continuous rehabilitation data and data support for adjusting individualized training programs.

[0059] In one embodiment, such as Figure 3 As shown, the processing layer 220 also includes an instruction decoder 222 and a voice synchronization controller 223.

[0060] The instruction decoder 222 is used to parse the preset training task configuration file to obtain the corresponding standard voice prompt data, standard voice lip-sync animation data, and standard upper limb movement demonstration animation data.

[0061] Indicatively, the instruction decoder is primarily used to receive and parse a preset training task configuration file. For example, this training task configuration file can adopt a structured data format such as JSON or XML, and internally encapsulates the corresponding training task's action type, voice instruction text, lip-sync template path, and upper limb movement keyframe data. Specifically, by parsing the training task configuration file, the instruction decoder can extract standard voice prompt data, standard voice lip-sync animation data, and standard upper limb movement demonstration animation data corresponding to the current training task. Optionally, the standard voice prompt data, standard voice lip-sync animation data, and standard upper limb movement demonstration animation data are all structured or media-based resource files that can be directly used for subsequent playback or display.

[0062] The voice synchronization controller 223 is used to align standard voice prompt data, standard voice lip-sync animation data, and standard upper limb movement demonstration animation data along the timeline to obtain the demonstration content.

[0063] In a schematic representation, the voice synchronization controller is responsible for aligning three types of data along the timeline: standard voice prompt data, standard voice lip-sync animation data, and standard upper limb movement demonstration animation data. Specifically, it analyzes the timeline of the standard voice prompts and their corresponding lip-sync sequences, establishing a mapping relationship between lip-sync image frames and voice time points. Simultaneously, it adjusts the time points of each keyframe in the standard upper limb movement demonstration animation data to correspond with semantic nodes in the voice content, thereby achieving precise synchronization of voice, lip-sync, and movement in the time dimension.

[0064] In one embodiment, the system further includes an interaction layer; the interaction layer includes a display module and a configuration module;

[0065] The display module is used to instruct the patient's display terminal to alternately display the demonstration content and the virtual upper limb movement feedback image at preset time intervals.

[0066] The display module is responsible for coordinating the content playback logic on the patient's display terminal, ensuring a smooth presentation rhythm and appropriate visual feedback. To enhance the patient's perception of the affected side's movements, an alternating display mechanism is employed. During training, the display module controls the patient's terminal to alternately display presentation content and virtual upper limb movement feedback images at preset time intervals. The presentation content includes a set of standard voice prompts, standard lip-sync animations, and standard upper limb movement animations corresponding to a training movement, guiding the patient's understanding of the training task. The virtual upper limb movement feedback images visually replace the movement of the affected upper limb, thereby inducing active activation and reintegration. Optionally, this alternating display strategy can be based on a fixed duration or dynamically adjusted according to training rhythm parameters.

[0067] The configuration module is used to respond to the obtained training action type, number of training actions, and training action rhythm, call the corresponding training action and voice joint template, and generate a training task configuration file.

[0068] The configuration module is primarily used to respond to training parameters input from the host system or the therapist's end, including information such as the type of training movement, the number of repetitions, and the training rhythm. For example, a therapist can preset a training task objective as 5 fist clenchings, each lasting 1 second, with the movement type being a basic daily life action. Upon receiving the training task objective, the configuration module will retrieve the corresponding training movement and voice joint template from the system resource library. This template includes standard voice prompts, standard lip-sync demonstration data, and standard upper limb movement keyframe data, and generates a complete time control table based on the movement rhythm parameters, resulting in a structured training task configuration file.

[0069] In one embodiment, the evaluation layer includes a motion evaluation module, a language evaluation module, and a feedback module;

[0070] The motion assessment module is used to analyze the consistency of motion amplitude between real-time motion images and corresponding standard upper limb motion demonstration animations to obtain the motion completion rate.

[0071] The motion assessment module analyzes the consistency of motion amplitude between real-time motion images captured during a patient's training task and corresponding standard upper limb movement demonstration animations. Specifically, the system extracts skeletal key point information from the motion images based on a posture recognition algorithm and constructs a temporal skeleton vector sequence. Simultaneously, it calls a preset skeleton trajectory template corresponding to the standard upper limb movement demonstration animation. The two are analyzed for similarity using methods such as spatial trajectory fitting and posture angle comparison. Optionally, multiple motion consistency indicators are introduced, including the correlation of joint angle change trends, the overlap of spatial trajectories, and the alignment of start and end postures, and these are integrated into a unified motion completion score to quantify the patient's motion execution effect.

[0072] The language evaluation module is used to analyze the consistency between the voiceprint of the speech data and the corresponding standard speech prompts to obtain the accuracy of the speech imitation.

[0073] The language assessment module analyzes the voiceprint consistency between the patient's spoken data and standard speech prompts. Illustratively, the system uses MFCC as the feature extraction method, combined with the DTW algorithm, to match the variation patterns of speech data in the time and frequency domains, thereby calculating the similarity between the patient's imitated speech and standard speech. Optionally, this module can further evaluate parameters such as pronunciation rhythm, intelligibility, and prosody, and comprehensively generate a language imitation accuracy score, reflecting the patient's auditory comprehension and pronunciation execution ability of language training instructions.

[0074] The feedback module is used to generate a comprehensive rehabilitation progress report based on the degree of action completion and the accuracy of language imitation.

[0075] The feedback module, as the output unit of the assessment layer, is responsible for receiving the result data from the motor assessment module and the language assessment module. Based on the set fusion weight model, it performs weighted calculations on the degree of motor completion and the accuracy of language imitation, generating a comprehensive rehabilitation progress report for the training cycle. This report may include the completion quality score of each round of training tasks, the time progress curve, and trend graphs of motor and speech sub-items, which are used to intuitively display the trajectory of changes in the patient's rehabilitation status.

[0076] In one embodiment, the consistency of the motion amplitude between real-time motion images and corresponding standard upper limb movement demonstration animations is analyzed to obtain the motion completion rate, including:

[0077] S11. Using a pose estimation algorithm based on human key point detection, extract the skeletal point sequences from the real-time motion images of the healthy upper limb and the affected upper limb to obtain the skeletal point sequences of the healthy upper limb and the affected upper limb.

[0078] The system utilizes a pose estimation algorithm based on human keypoint detection to process real-time motion images during training. Specifically, it identifies major skeletal points in each frame of the image, including the shoulder, elbow, wrist, and finger joints, and constructs a complete three-dimensional or two-dimensional skeletal point sequence trajectory. The system performs the above recognition operation on the real-time motion images of the patient's unaffected upper limb and the affected upper limb, respectively, thereby obtaining two sets of data: the skeletal point sequence of the unaffected upper limb and the skeletal point sequence of the affected upper limb.

[0079] S12. Compare the skeletal point sequence of the healthy upper limb with the standard skeletal point motion trajectory corresponding to the standard upper limb movement demonstration animation to obtain the movement execution correction gap.

[0080] The system compares the sequence of skeletal points on the healthy side with the standard skeletal point trajectories in the standard upper limb movement demonstration animation corresponding to the current task. It analyzes whether the patient's movements on the healthy side are consistent with the system's preset standard movements when performing the training task. If discrepancies exist, such as differences in movement angle, amplitude, or speed compared to the standard trajectory, it indicates a deviation in the patient's execution. The system defines this deviation as the movement execution correction gap and records it as a calibration factor for subsequent processing.

[0081] S13. Compare the skeletal point sequences of the healthy upper limb and the affected upper limb to obtain the difference in action execution.

[0082] The system compares the skeletal point sequences of the healthy upper limb with those of the affected upper limb to analyze whether the patient's affected limb reproduces the trajectory of the healthy side's movement under mirror-induced induction or active attempt. This comparison is achieved by calculating indicators such as the spatial offset of key point trajectories, the cosine similarity of joint angle changes, and the matching degree of velocity curves, outputting a set of metrics called the movement execution gap, which represents the degree to which the affected side's movement follows the movement of the healthy side.

[0083] S14. Adjust the action execution gap using the action execution correction factor to obtain the action completion rate.

[0084] The action execution correction factor originates from the error between the previous step on the healthy side and the standard action, and can be considered as the deviation of the patient's healthy side from the standard template during execution. The system uses this correction factor as a weighting factor to dynamically adjust the action execution gap between the healthy and affected sides, thereby avoiding misjudgments of the affected side's action due to deviations inherent in the healthy side's action. The comprehensive deviation value obtained after this correction serves as the final action completion index, used to quantify the quality of the patient's imitation of the target action on the affected side.

[0085] The aforementioned motor assessment module can take into account both the patient's subjective execution ability and actual imitation level, and has individual adaptability and assessment fairness. It is especially suitable for stroke hemiplegic patients in the early stage of rehabilitation when their movements are unstable and the healthy side has obvious compensation, providing more objective quantitative feedback.

[0086] In one embodiment, trajectory comparison corresponds to the following steps:

[0087] S21. The difference in the skeletal point sequence is evaluated based on the angle deviation, path similarity and velocity curve matching index to obtain deviation data.

[0088] The system evaluates the differences in skeletal point sequences based on three key indicators: angle deviation, path similarity, and velocity curve matching. Angle deviation measures the difference in angle change of corresponding key skeletal points in each frame, reflecting whether the movement is standard. Path similarity assesses the consistency of the overall movement trend by registering the spatial paths of key point trajectories in the time series. Velocity curve matching determines the coordination of the speed and rhythm of the movement, reflecting the difficulty the patient faces in performing the action.

[0089] S22. Calculate the deviation data using the cosine similarity of the pose vectors to obtain the skeletal point sequence difference.

[0090] Furthermore, cosine similarity of posture vectors is introduced as a comprehensive similarity calculation method. Specifically, the system treats the vector formed by the skeletal point sequence at a certain moment as a posture vector, and calculates the cosine similarity between the healthy side sequence and the target sequence (i.e., the standard movement or the affected side movement) to measure their consistency in spatial direction. This method is insensitive to amplitude differences but highly sensitive to posture direction, making it particularly suitable for analyzing the phenomenon of correct form but directional deviation in patient movements.

[0091] By calculating the cosine similarity of the pose vectors frame by frame and combining it with the weighted normalization of the above three-dimensional indicators, the system can generate a skeletal point sequence difference score for the entire action, which is the comprehensive quantitative result of trajectory comparison.

[0092] In one embodiment, analyzing the consistency between the voiceprint of the speech data and the corresponding standard speech prompt to obtain the accuracy of language imitation includes:

[0093] S31. Extract the Mel frequency cepstral coefficient features from the speech data to obtain speech features.

[0094] Indicatively, after receiving the speech data collected during training, the system can preprocess the raw speech signal, including basic operations such as denoising, endpoint detection, and frame segmentation. Further, the system performs MFCC feature extraction on the speech signal. This algorithm is widely used in speech recognition and speaker identification, effectively capturing the phonetic features of speech in both time and frequency dimensions. Specifically, the system converts the speech signal into a series of short frames. Each frame is Fourier transformed and mapped to the Mel frequency domain. Then, through logarithmic compression and discrete cosine transform, a compact and expressive MFCC feature sequence is finally obtained.

[0095] S32. Based on dynamic time warping, the speech features are matched with the audio features corresponding to the standard speech prompts to obtain the speech imitation accuracy.

[0096] The system retrieves the MFCC features of the corresponding audio samples from the standard speech prompts and uses the DTW algorithm to perform non-linear matching between the patient's speech features and the standard speech features. DTW is particularly suitable for handling speech comparison problems with inconsistent speaking speeds or rhythms, and can find the optimal alignment path on misaligned time axes. By calculating the cumulative distance between the two feature sequences on the matching path, the system can quantify the similarity between them and derive a speech imitation accuracy score. Optionally, the speech imitation accuracy score can be further mapped to a percentage or grade system as feedback on the patient's pronunciation quality, speech comprehension, and speech control abilities. The system can also construct hierarchical scoring rules based on different training tasks to adapt to the needs of different stages of language rehabilitation.

[0097] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0098] Based on the same inventive concept, this application also provides a method for implementing the above-mentioned voice and hand movement synchronized mirror training system. The solution provided by this method is similar to the implementation described in the above system. Therefore, the specific limitations in one or more embodiments of the voice and hand movement synchronized mirror training method provided below can be found in the limitations of the voice and hand movement synchronized mirror training system described above, and will not be repeated here.

[0099] In one exemplary embodiment, such as Figure 4 As shown, a method for synchronized mirror training of speech and hand movements is provided, including:

[0100] S401. In response to the training start command, send the demonstration content to the patient display.

[0101] S402. Real-time capture of motion images and voice data of the patient performing the demonstration content; real-time motion images include real-time motion images of the unaffected upper limb and the affected upper limb.

[0102] S403. Mirror the real-time motion image of the unaffected upper limb to generate a virtual upper limb motion feedback image, and send the virtual upper limb motion feedback image to the patient's display terminal.

[0103] S404. By analyzing real-time motion images and corresponding standard upper limb movement demonstration animations, as well as the matching degree between voice data and corresponding standard voice prompts, a comprehensive rehabilitation progress report is generated.

[0104] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0105] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0106] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0107] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A voice and hand movement synchronized mirror training system, characterized in that, The system includes: an information acquisition layer, a processing layer, and an evaluation layer; the processing layer includes a mirror generation unit. The information acquisition layer is used to respond to the training start command and capture real-time motion images and voice data of the patient during training; the training start command is used to instruct the patient's display terminal to execute the acquired demonstration content; the demonstration content includes standard voice prompts for playback, and standard voice lip-sync animation and standard upper limb movement demonstration animation for display; the real-time motion images include real-time motion images of the unaffected upper limb and real-time motion images of the affected upper limb. The mirror generation unit is used to mirror and flip the real-time motion image of the unaffected upper limb to generate a virtual upper limb motion feedback image, and send the virtual upper limb motion feedback image to the patient display terminal. The assessment layer is used to analyze the real-time motion images and the corresponding standard upper limb movement demonstration animations, as well as the matching degree between the voice data and the corresponding standard voice prompts, and to generate a comprehensive rehabilitation progress report.

2. The system according to claim 1, characterized in that: The processing layer also includes an instruction decoder and a voice synchronization controller; The instruction decoder is used to parse the preset training task configuration file to obtain the corresponding standard voice prompt data, standard voice lip-sync animation data and standard upper limb movement demonstration animation data. The voice synchronization controller is used to align the standard voice prompt data, the standard voice lip-sync animation data, and the standard upper limb movement demonstration animation data along the time axis to obtain the demonstration content.

3. The system according to claim 1, characterized in that: The system also includes an interaction layer; the interaction layer includes a display module and a configuration module. The display module is used to instruct the patient display terminal to alternately display the demonstration content and the virtual upper limb motion feedback image at preset time intervals; The configuration module is used to respond to the acquired training action type, number of training actions, and training action rhythm, call the corresponding training action and voice joint template, and generate the training task configuration file.

4. The system according to claim 1, characterized in that: The evaluation layer includes a motion evaluation module, a language evaluation module, and a feedback module; The motion assessment module is used to analyze the consistency of the motion amplitude between the real-time motion image and the corresponding standard upper limb motion demonstration animation to obtain the motion completion rate. The language evaluation module is used to analyze the consistency between the voice data and the corresponding standard voice prompts to obtain the accuracy of language imitation. The feedback module is used to generate the comprehensive rehabilitation progress report based on the degree of action completion and the accuracy of language imitation.

5. The system according to claim 4, characterized in that, The analysis of the consistency of the motion amplitude between the real-time motion image and the corresponding standard upper limb movement demonstration animation yields the motion completion rate, including: Using a pose estimation algorithm based on human key point detection, the skeletal point sequences of the real-time motion images of the healthy upper limb and the affected upper limb are extracted to obtain the skeletal point sequences of the healthy upper limb and the affected upper limb. The motion trajectory of the healthy upper limb skeletal point sequence and the standard skeletal point corresponding to the standard upper limb movement demonstration animation are compared to obtain the movement execution correction gap. The trajectory of the healthy upper limb skeletal point sequence and the affected upper limb skeletal point sequence is compared to obtain the action execution gap. The action completion rate is obtained by adjusting the action execution gap using the action execution correction factor.

6. The system according to claim 5, characterized in that, The trajectory comparison corresponds to the following steps: The difference between the skeletal point sequences is evaluated based on the metrics of angle deviation, path similarity, and velocity curve matching to obtain deviation data; The deviation data is calculated using the cosine similarity of the pose vectors to obtain the skeletal point sequence difference.

7. The system according to claim 4, characterized in that, The analysis of the consistency between the voice data and the corresponding standard voice prompt to obtain the accuracy of language imitation includes: Mel frequency cepstral coefficient feature extraction is performed on the speech data to obtain speech features; Based on dynamic time warping, the speech features are matched with the audio features corresponding to the standard speech prompts to obtain the speech imitation accuracy.

8. A method for synchronized mirror training of speech and hand movements, characterized in that, The method includes: In response to the training start command, the demonstration content is sent to the patient's display terminal; Real-time motion images and voice data are captured when the patient performs the demonstration content; the real-time motion images include real-time motion images of the unaffected upper limb and real-time motion images of the affected upper limb; The real-time motion image of the unaffected upper limb is mirrored and flipped to generate a virtual upper limb motion feedback image, which is then sent to the patient's display terminal. By analyzing the real-time motion images and corresponding standard upper limb movement demonstration animations, as well as the matching degree between the voice data and the corresponding standard voice prompts, a comprehensive rehabilitation progress report is generated.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method of claim 8.

Images