Rehabilitation training scheme generation method and device based on multi-modal data fusion, computer equipment and medium
By using multimodal data fusion technology, video and electromyography data are used to assess the functional levels of the upper limbs and hands, and personalized training movements are accurately matched. This solves the problem of insufficient personalized recommendations in existing rehabilitation training systems and improves rehabilitation effects and the adaptability of remote training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-04-10
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing rehabilitation training systems cannot accurately match and recommend personalized training movements, resulting in poor rehabilitation outcomes and difficulty in providing timely feedback remotely.
By fusing multimodal data, video data and surface electromyography data are acquired to assess kinematic characteristics and muscle activation timing coordination. Combined with abnormal compensatory activity, the functional levels of the upper limb and hand are accurately assessed, and personalized training movements are matched and recommended from the action library.
It enables personalized training movement recommendations, improves the effectiveness of rehabilitation training, adapts to the needs of remote rehabilitation training, and reduces the subjectivity of manual intervention.
Smart Images
Figure CN121839012A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of smart medical treatment, and in particular to a rehabilitation training scheme generation method and device based on multi-modal data fusion, a computer device and a medium. BACKGROUND
[0002] Stroke is a common neurological disease that often leads to motor dysfunction of upper limbs and hands. Rehabilitation training is an important means to help patients recover upper limb and hand function and improve quality of life.
[0003] Rehabilitation training of upper limbs and hands needs to be personalized according to the specific situation of patients. Each patient has different conditions, degrees of motor function impairment, and rehabilitation goals, so a targeted rehabilitation training scheme needs to be developed.
[0004] Traditional rehabilitation training methods are usually developed by rehabilitation therapists according to experience, but this method has certain subjectivity and errors, and it is difficult to implement remotely or provide timely feedback. At the same time, due to the large number of patients and limited therapist resources, it is difficult to develop a fine rehabilitation training scheme for each patient.
[0005] In the prior art, in order to improve the standardization and efficiency of rehabilitation training, some rehabilitation training systems based on a pre-defined action library have appeared. Such systems usually obtain preliminary evaluation results from patient training videos, and then match and recommend training actions from the action library, so that accurate and personalized training actions cannot be matched and recommended, affecting the rehabilitation training effect. SUMMARY
[0006] Therefore, the embodiments of the present application provide a rehabilitation training scheme generation method based on multi-modal data fusion to solve the technical problem that accurate and personalized training actions cannot be matched and recommended in the prior art. The method comprises: During the execution of the evaluation action by the target object, video data and surface electromyography data of the target object are obtained; For each of the upper limb action, the hand action, and the upper limb and hand comprehensive action, kinematic characteristics of the action are obtained based on the video data, and muscle activation timing coordination and abnormal compensation activity degree of the action are obtained based on the surface electromyography data; Based on the kinematic characteristics, muscle activation timing coordination, and abnormal compensation activity degree of the upper limb action, the upper limb function level of the target object is evaluated, based on the kinematic characteristics, muscle activation timing coordination, and abnormal compensation activity degree of the hand action, the hand function level of the target object is evaluated, and based on the kinematic characteristics, muscle activation timing coordination, and abnormal compensation activity degree of the upper limb and hand comprehensive action, the upper limb and hand comprehensive function level of the target object is evaluated; According to the upper limb function level, the hand function level and the upper limb and hand comprehensive function level of the target object, a recommended training action is matched in the upper limb action sub-library, the hand action sub-library and the upper limb and hand comprehensive action sub-library in the action library.
[0007] The embodiment of the present application also provides a rehabilitation training scheme generation device based on multi-modal data fusion, to solve the technical problem that precise and personalized training actions cannot be matched and recommended in the prior art. The data acquisition module is configured to acquire video data and surface electromyography data of the target object in a process in which the target object performs the assessment action; The index acquisition module is configured to acquire kinematics characteristics of each of the upper limb action, the hand action and the upper limb and hand comprehensive action based on the video data, and acquire muscle activation timing coordination and abnormal compensation activity degree of the action based on the surface electromyography data; The level evaluation module is configured to evaluate the upper limb function level of the target object based on the kinematics characteristics, the muscle activation timing coordination and the abnormal compensation activity degree of the upper limb action, evaluate the hand function level of the target object based on the kinematics characteristics, the muscle activation timing coordination and the abnormal compensation activity degree of the hand action, and evaluate the upper limb and hand comprehensive function level of the target object based on the kinematics characteristics, the muscle activation timing coordination and the abnormal compensation activity degree of the upper limb and hand comprehensive action. The action recommendation module is configured to match and recommend a training action that is suitable for the target object according to the upper limb function level, the hand function level and the upper limb and hand comprehensive function level of the target object in the upper limb action sub-library, the hand action sub-library and the upper limb and hand comprehensive action sub-library in the action library.
[0008] The embodiment of the present application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements any of the rehabilitation training scheme generation methods based on multi-modal data fusion described above when the computer program is executed, to solve the technical problem that precise and personalized training actions cannot be matched and recommended in the prior art.
[0009] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program for implementing any of the rehabilitation training scheme generation methods based on multi-modal data fusion described above, to solve the technical problem that precise and personalized training actions cannot be matched and recommended in the prior art.
[0010] Compared with the prior art, the beneficial effects that at least one of the above-mentioned technical solutions adopted in the embodiments of this specification can achieve include at least the following: It proposes to determine objective kinematic characteristics, muscle activation timing coordination, and abnormal compensatory activity based on multimodal real data such as video data and surface electromyography data; and to accurately and personally determine the upper limb function level, the hand function level, and the comprehensive upper limb and hand function level of each target object based on the objective kinematic characteristics, muscle activation timing coordination, and abnormal compensatory activity level. Furthermore, based on the accurate and personalized upper limb function level, the hand function level, and the comprehensive upper limb and hand function level, it matches recommended and suitable training movements in the corresponding action sub-library, enabling accurate and personalized matching of recommended and suitable training movements, thereby improving the rehabilitation training effect. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart of a method for generating a rehabilitation training program based on multimodal data fusion, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating an evaluation, push scheme, and database update process provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating a method for determining a functional level, provided by an embodiment of the present invention. Figure 4 This is a structural block diagram of a computer device provided in an embodiment of the present invention; Figure 5 This is a structural block diagram of a rehabilitation training program generation device based on multimodal data fusion provided in an embodiment of the present invention. Detailed Implementation
[0013] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0014] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0015] In this embodiment of the invention, a method for generating rehabilitation training programs based on multimodal data fusion is provided, such as... Figure 1 As shown, the method includes: Step S101: During the process of the target object performing the evaluation action, acquire the video data and surface electromyography data of the target object; Step S102: For each movement in the upper limb movement, hand movement, and combined upper limb and hand movement, obtain the kinematic characteristics of the movement based on the video data, and obtain the muscle activation timing coordination and abnormal compensatory activity of the movement based on the surface electromyography data. Step S103: Based on the kinematic characteristics of upper limb movements, muscle activation timing coordination, and abnormal compensatory activity, assess the upper limb function level of the target object; based on the kinematic characteristics of hand movements, muscle activation timing coordination, and abnormal compensatory activity, assess the hand function level of the target object; based on the kinematic characteristics of combined upper limb and hand movements, muscle activation timing coordination, and abnormal compensatory activity, assess the combined upper limb and hand function level of the target object. Step S104: Based on the target object's upper limb function level, hand function level, and combined upper limb and hand function level, match and recommend suitable training movements from the upper limb movement sub-library, hand movement sub-library, and combined upper limb and hand movement sub-library in the movement library.
[0016] In practice, the above assessment actions can be a set of standardized assessment actions (consisting of three specific sets of upper limb actions A, hand actions B, and combined upper limb and hand actions C) to initialize and periodically assess the target object's upper limb function, hand function, and combined upper limb and hand motor function respectively.
[0017] In specific implementation, such as Figure 2As shown, assessment data is collected during the target subject's performance of the assessment actions. For example, video data of the target subject performing the assessment actions and rehabilitation training can be simultaneously acquired via a camera. Based on the video data, kinematic features related to upper limb and hand movements, such as range of motion, fluency, and speed, can be extracted. Surface electromyography (sEMG) sensors can be used to simultaneously acquire surface electromyography signal data of the main target muscle groups during the target subject's performance of the assessment actions, for analysis of neuromuscular control efficacy (i.e., muscle activation timing coordination and abnormal compensatory activity).
[0018] In specific implementation, such as Figure 2 As shown, after obtaining the assessment data, functional level assessment (or immediate grading) can be performed based on the assessment data. To further accurately assess functional level, a method considering muscle activation timing coordination is proposed for assessment, and a method for quantitatively calculating muscle activation timing coordination is further proposed. For example, the muscle activation timing coordination of the movement is obtained based on the surface electromyography data, including: Extract the activation time sequence of the preset target muscle (such as agonist, antagonist and synergist) corresponding to the action to form a first vector; use the standard activation time sequence of the preset target muscle as a second vector; specifically, upper limb actions and hand actions can correspond to different target muscles.
[0019] Calculate the minimum alignment distance between the first vector and the second vector, and calculate the temporal disorder of the first vector and the second vector; The muscle activation timing coordination is calculated based on the minimum alignment distance and the temporal disorder.
[0020] Specifically, the Dynamic Time Warping (DTW) algorithm can be used to calculate the minimum alignment distance between sequence P (i.e., the first vector) and R (i.e., the second vector). : .
[0021] In specific implementation, the temporal disorder of the first vector and the second vector is calculated, including: The temporal disorder value for each muscle pair is calculated using the following formula:
[0022] Among them, the Let (i,j) be the temporal disorder value for muscle pair (i,j). This represents the actual activation time of the i-th target muscle in the first vector. Let be the actual activation time of the j-th target muscle in the first vector. The standard activation time of the i-th target muscle in the second vector is given. The standard activation time is the j-th target muscle in the second vector; specifically, the same target muscles are ordered in the same order in the first and second vectors. In a muscle pair (i,j), i and j can be two adjacent or non-adjacent target muscles in the first vector, representing a key muscle combination with antagonistic or synergistic functional relationships.
[0023] Based on the temporal disorder values of all muscle pairs, the temporal disorder degree of the first vector and the second vector is calculated. Specifically, the mean or maximum value of the temporal disorder values of all muscle pairs can be used as the temporal disorder degree of the first vector and the second vector. For example, the mean value of the temporal disorder values of all muscle pairs can be used as the temporal disorder degree of the first vector and the second vector. : M represents the total number of muscle pairs.
[0024] In specific implementation, the muscle activation timing coordination is calculated based on the minimum alignment distance and the temporal disorder, including: The numerical value of the muscle activation timing coordination is calculated using the following formula: in, This represents the numerical value of the muscle activation timing coordination. The maximum value of the predefined DTW distance. The minimum alignment distance, , These are weighting coefficients. The time sequence disorder is described.
[0025] In practice, after obtaining the numerical value of muscle activation timing coordination, it can be divided into different sub-levels based on the magnitude of the value. For example, based on the numerical value of muscle activation timing coordination... It can be divided into 5 sub-levels, L1-L5: L5: ≤0.1 (highly coordinated timing) indicates excellent / normal neuromuscular control with high efficiency and precision and no unnecessary energy consumption. The activation sequence of agonist, antagonist, and synergist muscles is completely correct, with a timing error of <50ms.
[0026] L4:0.1< ≤0.25 indicates good performance; control function is good, with slight flaws that do not affect the quality of motion completion. Timing is basically correct, with slight delays / advances (50ms ≤ timing error ≤ 100ms). Cooperative contraction may be slightly more or less.
[0027] L3:0.25 ≤0.45 indicates moderate / fair performance, with significant abnormalities in neural control, but the ability to mimic the appearance of the movement is still possible, though inefficient. Timing is disordered. For example, antagonist muscles fail to relax in a timely manner, or the activation sequence of synergist muscles is incorrect (100ms < timing error ≤ 200ms). Inaccurate intensity control.
[0028] L2:0.45< A score ≤0.7 indicates poor performance, with difficulty in completing the movement, distorted form, and heavy reliance on compensation. Timing is severely disrupted. The muscle activation sequence is almost impossible to identify as a normal pattern (timing error >200ms).
[0029] L1: A score >0.7 (extremely disordered timing) indicates severe damage, an inability to initiate or complete the target movement independently, and no effective timing pattern. Muscle activity is random, scattered, or completely silent, and the activation sequence related to the movement intention cannot be identified.
[0030] In practical implementation, to further accurately assess functional levels, a method considering abnormal compensatory activity is proposed for evaluating functional levels. This leads to a method for quantifying abnormal compensatory activity, such as obtaining the abnormal compensatory activity of the movement based on the surface electromyography data, including: The surface electromyography signals of the target muscle group and non-target muscle group of the movement are integrated during the movement cycle to obtain the target muscle energy and non-target muscle energy. The baseline compensatory energy ratio is calculated based on the ratio of the target muscle energy to the non-target muscle energy. The surface electromyography signals of the non-target muscle group are weighted and integrated within the movement cycle to obtain the abnormal compensation level energy. The abnormal compensation level ratio is calculated based on the ratio of the abnormal compensation level energy to the target muscle energy. The value of abnormal compensation activity is calculated based on the basic compensation energy ratio and the proportion of abnormal compensation level.
[0031] Specifically, target muscle energy for: ,in, For the first Surface electromyographic signals of bulk target muscles The total number of target muscles in the target muscle group; Non-target muscle energy for: ,in, For the first Surface electromyography (EMG) signals of a non-target muscle. This represents the total number of non-target muscles within the non-target muscle group. Basic compensatory energy ratio for: .
[0032] Abnormal compensatory level energy for: ,in, For weighted weights, percentage of abnormal compensation levels for: .
[0033] In specific implementation, the value of abnormal compensation activity is calculated based on the basic compensation energy ratio and the proportion of abnormal compensation level, including: The value of the abnormal compensatory activity is calculated using the following formula:
[0034] in, This represents the numerical value of abnormal compensatory activity. The aforementioned basic compensatory energy ratio, This represents the percentage of the abnormal compensation level. , Energy at abnormal compensatory levels. For target muscle energy, α and β are weighting coefficients.
[0035] In practice, after obtaining the numerical value of abnormal compensatory activity, it can be divided into different sub-levels based on the magnitude of the value. For example, based on the numerical value of abnormal compensatory activity, it can be divided into 5 sub-levels, L1-L5: L5: ≤0.1 (no or slight compensation) indicates no or slight compensation. Activation level of non-target muscles <10% (relative to the target agonist muscle). Good movement isolation.
[0036] L4:0.1< ≤0.2 indicates mild compensation. At the maximum difficulty of the movement, the activation of some non-target muscles increases (10% ~ 20%).
[0037] L3:0.2< A value ≤0.4 indicates significant compensation. Multiple non-target muscles are involved in the force exertion (20% ~ 40%), and a fixed abnormal compensation pattern begins to form.
[0038] L2:0.4< A value ≤0.6 indicates severe compensation. Compensating muscles become the main force in completing the movement (activation level >40%), while the activation of the target muscle is inhibited.
[0039] L1: A score >0.6 (completely compensatory) indicates either complete compensatory dominance or ineffective systemic activation. When the patient attempts to move, completely unrelated muscle groups are activated.
[0040] In practice, kinematic characteristics (K) can also be divided into five sub-levels, L1-L5: L5: Near normal, with speed and accuracy errors less than 5%.
[0041] L4: The range of motion of the joint is 80% greater than the standard range. For example, the upper limb can touch the head, the hand can grasp, and the upper limb can lift and grasp the ball completely, but the speed and accuracy are poor.
[0042] L3: The range of motion of the joint is 50%-80% of the standard range. For example, the upper limb can touch the head most of the time, the hand can grasp most of the time, and the upper limb can lift most of the time and grasp the ball. However, if the ball is touched, it cannot be grasped.
[0043] L2: The range of motion of the joint is less than 50% of the standard range. For example, the upper limb can partially complete the action of touching the head, the hand can be slightly flexed, and the upper limb can partially lift and grasp the ball.
[0044] L1: No active movement, range of motion less than 5%.
[0045] In practice, after calculating the muscle activation timing coordination and abnormal compensatory activity, the results can be applied based on kinematic characteristics, muscle activation timing coordination, and abnormal compensatory activity (i.e., Figure 2 The "kinematic characteristics combined with neuromuscular control efficacy" model shown separately assesses upper limb function, hand function, and combined upper limb and hand function. For example, ... Figure 3 As shown, based on the kinematic characteristics of upper limb movements, muscle activation timing coordination, and abnormal compensatory activity, the upper limb function level of the target subject is assessed; based on the kinematic characteristics of hand movements, muscle activation timing coordination, and abnormal compensatory activity, the hand function level of the target subject is assessed; and based on the kinematic characteristics of combined upper limb and hand movements, muscle activation timing coordination, and abnormal compensatory activity, the combined upper limb and hand function level of the target subject is assessed, including: For each function in the upper limb function, hand function, and combined upper limb and hand function, the kinematic feature sub-level, muscle activation timing coordination sub-level, and abnormal compensatory activity sub-level are obtained. For example, the kinematic feature sub-level of the function is determined based on the kinematic characteristics of the corresponding action, the muscle activation timing coordination sub-level of the function is determined based on the value of the muscle activation timing coordination of the corresponding action, and the abnormal compensatory activity sub-level of the function is determined based on the value of the abnormal compensatory activity of the corresponding action. Determine if two or more of the following sub-levels—the kinematic feature sub-level, the muscle activation timing coordination sub-level, and the abnormal compensatory activity sub-level—are at the lowest level (e.g., L1 sub-level): If yes, then determine that functional level is the lowest level (i.e., L1 level); if not, determine if the abnormal compensatory activity sub-level is the lowest level: If yes, determine the upper limit of the first level (e.g., L1 level). Figure 3 The L2 level is shown, and the level that appears most frequently among the kinematic feature sub-level, the muscle activation timing coordination sub-level, and the abnormal compensatory activity sub-level is taken as the candidate level. If the three sub-levels are different, the lowest level among them is taken as the candidate level. The lower of the candidate level and the first level upper limit is determined as the functional level of the function; otherwise, the second level upper limit is determined (e.g., L2 level). Figure 3 The L5 level is shown, and the level that appears most frequently among the kinematic feature sub-level, the muscle activation timing coordination sub-level, and the abnormal compensatory activity sub-level is taken as the candidate level. If the three sub-levels are different, the lowest level among them is taken as the candidate level. The lower level between the candidate level and the upper limit of the second level is determined as the functional level of the function (that is, the functional level of the function can be divided into 5 levels such as L1-L5).
[0046] In specific implementation, after obtaining the upper limb functional level, the hand functional level, and the combined upper limb and hand functional level, as follows: Figure 2 As shown, training movements can be matched and recommended from the upper limb movement sub-library, hand movement sub-library, and upper limb and hand comprehensive movement sub-library in the movement library (i.e., the pre-built library) based on the upper limb function level, the hand function level, and the upper limb and hand comprehensive function level.
[0047] In practice, each training movement in the upper limb movement sub-library, hand movement sub-library, and upper limb and hand integrated movement sub-library of the above-mentioned movement library also has its own level (such as five levels, L1-L5) and three dimensions of sub-levels: kinematic characteristics, muscle activation timing coordination, and abnormal compensatory activity. The evaluation principle of the level and sub-level of the training movement is similar to the principle of evaluating the upper limb function level, the hand function level, and the upper limb and hand integrated function level and the sub-levels of each dimension of the target object.
[0048] Specifically, during the matching and recommendation of training movements, based on the functional level of each function and / or the kinematic characteristics of each function, the three-dimensional sub-levels of muscle activation timing coordination and abnormal compensatory activity, the system can automatically match movements to the patient (i.e. the target subject) according to priority at the corresponding level of each sub-database (including three sub-databases: upper limb, fine motor, and comprehensive upper limb) and generate a highly personalized and immediate rehabilitation training plan.
[0049] For example, there are upper limb training sub-libraries (i.e., upper limb movement sub-libraries, also known as Library A), hand fine motor training sub-libraries (i.e., hand movement sub-libraries, also known as Library B), and upper limb comprehensive training sub-libraries (i.e., upper limb and hand comprehensive movement sub-libraries, also known as Library C).
[0050] Each sub-library includes five levels, L1-L5: upper limb training sub-library L1-L5 (also known as A-library L1-L5), hand fine motor training sub-library L1-L5 (also known as B-library L1-L5), and upper limb comprehensive training sub-library L1-L5 (also known as C-library L1-L5).
[0051] Furthermore, each training action instance in each training sub-library is labeled with three dimensions: kinematic feature K, muscle activation timing coordination T, and abnormal compensatory activity C1.
[0052] The priority is as follows: When the system performs action matching based on the patient's three-dimensional sub-level labels, it follows these priority rules: Perfect match: Training actions with completely identical levels in the three dimensions of K, T, and C1; Core dimension matching: K and C1 dimensions are matched, and training actions with a difference of no more than 1 level in the T dimension. Basic Dimension Matching: K-dimensional matching, training actions where other dimensions differ by no more than 1 level. Intra-level matching: Other training actions within the same functional level.
[0053] Example: The patient's upper limb function assessment result is determined to be A-L3 (K-L3, T-L2, C1-L3). The system will automatically match 1-2 training movements from the training library A-L3 level movement library according to this level. If there are no movements in the pre-built training library that fully match the patient's three-dimensional upper limb level (K-L3, T-L2, C1-L3), then the level movement will be selected according to the following steps.
[0054] Example: An example of an action is "touching the mouth with the upper limb from a natural hanging position". This action belongs to the upper limb training sub-library L3. In addition, this action also has the tags "kinematic characteristics L3", "muscle activation timing coordination L2", and "abnormal compensatory activity L3", that is, this action is A-L3 (K-L3, T-L2, C1-L3). In practice, patients can execute the generated personalized training program under the guidance of a therapist in a clinical setting or independently in a home setting.
[0055] In a clinical setting, in addition to performing the training exercises generated by the system, therapists may add new training exercises based on the patient's condition. During this process, video data and surface electromyography (EMG) data of the training exercises can be continuously and synchronously collected using the aforementioned cameras, surface EMG sensors, and other devices.
[0056] After the assessment and recommended training movements are completed, the system can display the patient's overall functional level in three areas (upper limb A, hand B, and combined upper limb and hand movement C), and list the sub-levels (K, T, C1) of each area in three dimensions, in preparation for subsequent training program matching and automatic updates of the training library.
[0057] For example, the patient assessment results may be: overall functional level A-L3, B-L4, C-L3, where the sub-levels of each dimension under A are K-L3, T-L2, C1-L3; the sub-levels of each dimension under B are K-L4, T-L4, C1-L3; and the sub-levels of each dimension under C are K-L3, T-L3, C1-L2.
[0058] The patient's assessment results displayed in the system are: A-L3 (K-L3, T-L2, C1-L3), B-L4 (K-L4, T-L4, C1-L3), C-L3 (K-L3, T-L3, C1-L2).
[0059] In practical implementation, to further improve the accuracy of matched recommended training movements, the aforementioned movement library can be dynamically and automatically enriched and structured, becoming a self-improving and adaptively evolving dynamic training library. It can automatically collect real training data from patients during rehabilitation, extract valuable training movements, and accurately classify and store them based on the patient's objective kinematic characteristics and neuromuscular function level (i.e., muscle activation timing coordination and abnormal compensatory activity). This constructs a living knowledge base originating from clinical practice and continuously optimized for clinical practice, laying a solid foundation for generating highly personalized rehabilitation training programs.
[0060] Specifically, such as Figure 2 As shown, the training library (i.e., the action library mentioned above) undergoes accompanying automatic updates: The training videos captured by the camera are input into a pre-trained rehabilitation training model (an AI model based on motion recognition). The model decomposes the continuous training videos into multiple discrete, standardized training movements (such as "raising the upper limb to touch the head", "grasping with the hand", "raising the upper limb and grasping the ball with the hand", etc.).
[0061] Machine learning and other technologies are used to classify the training movements decomposed above. First, the training movement is categorized into the corresponding training sub-library based on the training area. Then, it is associated with the determined functional level of the patient, and this is used as an index key to add it to the corresponding rehabilitation training library.
[0062] That is, after the training action is decomposed and identified by the model, the system will classify the training action into the corresponding level of the training sub-library based on the functional level obtained by the current evaluation. Each action instance in the training library has a quantitative label with three dimensions (K, T, C1).
[0063] The system categorizes training actions and can identify the main motion joints of the actions based on a computer vision model, according to the following rules: Movements involving the shoulder and elbow joints are categorized as Library A. When the movement involves finger joint bending and the range of motion is greater than 40%, it is classified as Category B. Other actions that involve both large joints of the upper limbs and fine motor skills of the hand are classified as Library C.
[0064] The training sub-libraries are the same as the training sub-libraries mentioned above, including upper limb training sub-libraries L1-L5 (also known as library A L1-L5), hand fine motor training sub-libraries L1-L5 (also known as library B L1-L5), and upper limb comprehensive training sub-libraries L1-L5 (also known as library C L1-L5).
[0065] For example: If an action instance is identified as being in the upper limb training sub-library, and the three dimensions of the upper limb L3 patient who performed this training action are kinematic characteristics L3, muscle activation timing coordination L2, and abnormal compensatory activity L3, then the action is classified as A-L3 (the patient is identified in the assessment step), and is labeled with "kinematic characteristics L3", "muscle activation timing coordination L2", and "abnormal compensatory activity L3", that is, the action is A-L3 (K-L3, T-L2, C1-L3).
[0066] Once the system identifies new training movements, it employs an automatic movement selection and manual supervision mechanism to ensure the quality and effectiveness of the training library. By continuously collecting new patient data, the movement examples in the training library are constantly enriched, covering a wider range of functional levels and movement variations, thus achieving automated improvement of the training library. During long-term rehabilitation training, patients are regularly assessed for effectiveness, and the assessment results are fed back into the database to facilitate optimization and adjustment of the rehabilitation training program.
[0067] Automatic action screening: Only when an action is consistently and repeatedly performed by multiple patients at the same functional level, and its kinematic / electromyographic characteristics are clearly clustered, can it be promoted to "standard action instance" and added to the core library.
[0068] Human supervision: The system provides a backend interface for rehabilitation therapists to review, approve, or reject new movements recommended by the system for inclusion in the database.
[0069] In this embodiment, a computer device is provided, such as... Figure 4 As shown, it includes a memory 401, a processor 402, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned method for generating any rehabilitation training program based on multimodal data fusion.
[0070] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.
[0071] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that executes any of the above-described methods for generating rehabilitation training programs based on multimodal data fusion.
[0072] Specifically, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.
[0073] Based on the same inventive concept, this invention also provides a rehabilitation training program generation device based on multimodal data fusion, as described in the following embodiments. Since the principle of the rehabilitation training program generation device based on multimodal data fusion is similar to that of the rehabilitation training program generation method based on multimodal data fusion, the implementation of the rehabilitation training program generation device based on multimodal data fusion can refer to the implementation of the rehabilitation training program generation method based on multimodal data fusion, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0074] Figure 5 This is a structural block diagram of a rehabilitation training program generation device based on multimodal data fusion according to an embodiment of the present invention, such as... Figure 5 As shown, it includes:
[0075] The data acquisition module 501 is used to acquire video data and surface electromyography data of the target object during the process of the target object performing the evaluation action; The indicator acquisition module 502 is used to acquire the kinematic characteristics of each movement in the upper limb movement, hand movement, and combined upper limb and hand movement based on the video data, and to acquire the muscle activation timing coordination and abnormal compensatory activity of the movement based on the surface electromyography data. The rating assessment module 503 is used to assess the upper limb function level of the target object based on the kinematic characteristics of upper limb movements, muscle activation timing coordination, and abnormal compensatory activity; to assess the hand function level of the target object based on the kinematic characteristics of hand movements, muscle activation timing coordination, and abnormal compensatory activity; and to assess the comprehensive upper limb and hand function level of the target object based on the kinematic characteristics of combined upper limb and hand movements, muscle activation timing coordination, and abnormal compensatory activity. The action recommendation module 504 is used to match and recommend suitable training actions from the upper limb action sub-library, hand action sub-library, and upper limb and hand comprehensive action sub-library in the action library according to the target object's upper limb function level, hand function level, and upper limb and hand comprehensive action sub-library.
[0076] In one embodiment, the indicator acquisition module is used to extract the activation time sequence of a preset target muscle corresponding to the action to form a first vector; use the standard activation time sequence of the preset target muscle as a second vector; calculate the minimum alignment distance between the first vector and the second vector, calculate the temporal disorder between the first vector and the second vector; and calculate the muscle activation temporal coordination based on the minimum alignment distance and the temporal disorder.
[0077] In one embodiment, the indicator acquisition module is used to calculate the temporal disorder value for each muscle pair using the following formula:
[0078] Among them, the Let (i,j) be the temporal disorder value for muscle pair (i,j). This represents the actual activation time of the i-th target muscle in the first vector. Let be the actual activation time of the j-th target muscle in the first vector. The standard activation time of the i-th target muscle in the second vector is given. Let be the standard activation time of the j-th target muscle in the second vector; calculate the temporal disorder of the first vector and the second vector based on the temporal disorder values of all muscle pairs.
[0079] In one embodiment, the indicator acquisition module is used to calculate the numerical value of the muscle activation timing coordination using the following formula: in, This represents the numerical value of the muscle activation timing coordination. The maximum value of the predefined DTW distance. The minimum alignment distance, , These are weighting coefficients. The time sequence disorder is described.
[0080] In one embodiment, the indicator acquisition module is configured to integrate the surface electromyography (EMG) signals of the target muscle group and non-target muscle group during the movement cycle to obtain target muscle energy and non-target muscle energy; calculate the baseline compensatory energy ratio based on the ratio of the target muscle energy to the non-target muscle energy; perform weighted integration of the EMG signals of the non-target muscle group during the movement cycle to obtain abnormal compensatory level energy; calculate the abnormal compensatory level percentage based on the ratio of the abnormal compensatory level energy to the target muscle energy; and calculate the abnormal compensatory activity value based on the baseline compensatory energy ratio and the abnormal compensatory level percentage.
[0081] In one embodiment, the indicator acquisition module is used to calculate the value of the abnormal compensatory activity degree using the following formula:
[0082] in, This represents the numerical value of abnormal compensatory activity. The aforementioned basic compensatory energy ratio, This represents the percentage of the abnormal compensation level. , Energy at abnormal compensatory levels. For target muscle energy, α and β are weighting coefficients.
[0083] In one embodiment, the rating assessment module is used to determine the kinematic feature sub-level of each function among upper limb function, hand function, and integrated upper limb and hand function, based on the kinematic characteristics of the corresponding movement; determine the muscle activation timing coordination sub-level of the corresponding movement based on the value of the muscle activation timing coordination; and determine the abnormal compensatory activity sub-level of the corresponding movement based on the value of the abnormal compensatory activity. It then determines whether two or more of the kinematic feature sub-level, the muscle activation timing coordination sub-level, and the abnormal compensatory activity sub-level are of the lowest level. If so, the function level is determined to be the lowest level; otherwise, it is determined to be lower. If the abnormal compensatory activity level is the lowest level, determine the upper limit of the first level and select the level that appears most frequently among the kinematic feature level, the muscle activation timing coordination level, and the abnormal compensatory activity level as the candidate level. The lower level between the candidate level and the upper limit of the first level is determined as the functional level of the function. If not, determine the upper limit of the second level and select the level that appears most frequently among the kinematic feature level, the muscle activation timing coordination level, and the abnormal compensatory activity level as the candidate level. The lower level between the candidate level and the upper limit of the second level is determined as the functional level of the function.
[0084] The embodiments of this invention achieve the following technical effects: They propose using multimodal real data such as video data and surface electromyography (EMG) data to determine objective kinematic characteristics, muscle activation timing coordination, and abnormal compensatory activity. Based on these objective kinematic characteristics, muscle activation timing coordination, and abnormal compensatory activity, they accurately and personally determine the upper limb function level, hand function level, and overall upper limb and hand function level for each target subject. Then, based on these accurate and personalized upper limb function levels, hand function levels, and overall upper limb and hand function levels, they match recommended and suitable training movements in the corresponding action sub-library. This allows for accurate and personalized matching of recommended and suitable training movements, thereby improving the effectiveness of rehabilitation training.
[0085] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.
[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for generating rehabilitation training programs based on multimodal data fusion, characterized in that, include: During the process of the target object performing the assessment action, video data and surface electromyography data of the target object are acquired; For each movement in the upper limb movement, hand movement, and combined upper limb and hand movement, the kinematic characteristics of the movement are obtained based on the video data, and the muscle activation timing coordination and abnormal compensatory activity of the movement are obtained based on the surface electromyography data. The upper limb function level of the target subject is assessed based on the kinematic characteristics of upper limb movements, muscle activation timing coordination, and abnormal compensatory activity. The hand function level of the target subject is assessed based on the kinematic characteristics of hand movements, muscle activation timing coordination, and abnormal compensatory activity. The comprehensive upper limb and hand function level of the target subject is assessed based on the kinematic characteristics of combined upper limb and hand movements, muscle activation timing coordination, and abnormal compensatory activity. Based on the target object's upper limb function level, hand function level, and combined upper limb and hand function level, recommended and suitable training movements are matched from the upper limb movement sub-library, hand movement sub-library, and combined upper limb and hand movement sub-library in the movement library.
2. The method as described in claim 1, characterized in that, Based on the surface electromyography data, the timing coordination of muscle activation during the movement is obtained, including: Extract the activation time sequence of the preset target muscle corresponding to the action to form a first vector; use the standard activation time sequence of the preset target muscle as a second vector; Calculate the minimum alignment distance between the first vector and the second vector, and calculate the temporal disorder of the first vector and the second vector; The muscle activation timing coordination is calculated based on the minimum alignment distance and the temporal disorder.
3. The method as described in claim 2, characterized in that, Calculate the temporal disorder of the first vector and the second vector, including: The temporal disorder value for each muscle pair is calculated using the following formula: Among them, the Let (i,j) be the temporal disorder value for muscle pair (i,j). This represents the actual activation time of the i-th target muscle in the first vector. Let be the actual activation time of the j-th target muscle in the first vector. The standard activation time of the i-th target muscle in the second vector is given. The standard activation time of the j-th target muscle in the second vector; Calculate the temporal disorder degree of the first vector and the second vector based on the temporal disorder values of all muscle pairs.
4. The method as described in claim 2, characterized in that, The muscle activation temporal coordination is calculated based on the minimum alignment distance and the temporal disorder, including: The numerical value of the muscle activation timing coordination is calculated using the following formula: in, This represents the numerical value of the muscle activation timing coordination. The maximum value of the predefined DTW distance. The minimum alignment distance, , These are weighting coefficients. The time sequence disorder is described.
5. The method as described in claim 1, characterized in that, The abnormal compensatory activity level of the movement is obtained based on the surface electromyography data, including: The surface electromyography signals of the target muscle group and non-target muscle group of the movement are integrated during the movement cycle to obtain the target muscle energy and non-target muscle energy. The baseline compensatory energy ratio is calculated based on the ratio of the target muscle energy to the non-target muscle energy. The surface electromyography signals of the non-target muscle group are weighted and integrated within the movement cycle to obtain the abnormal compensation level energy. The abnormal compensation level ratio is calculated based on the ratio of the abnormal compensation level energy to the target muscle energy. The value of abnormal compensation activity is calculated based on the basic compensation energy ratio and the proportion of abnormal compensation level.
6. The method as described in claim 5, characterized in that, Based on the basic compensation energy ratio and the proportion of abnormal compensation levels, the value of abnormal compensation activity is calculated, including: The value of the abnormal compensatory activity is calculated using the following formula: in, This represents the numerical value of abnormal compensatory activity. The aforementioned basic compensatory energy ratio, This represents the percentage of the abnormal compensation level. , Energy at abnormal compensatory levels. For target muscle energy, α and β are weighting coefficients.
7. The method according to any one of claims 1 to 6, characterized in that, Based on the kinematic characteristics of upper limb movements, muscle activation timing coordination, and abnormal compensatory activity, the upper limb function level of the target subject is assessed. Based on the kinematic characteristics of hand movements, muscle activation timing coordination, and abnormal compensatory activity, the hand function level of the target subject is assessed. Based on the kinematic characteristics of combined upper limb and hand movements, muscle activation timing coordination, and abnormal compensatory activity, the combined upper limb and hand function level of the target subject is assessed, including: For each function among upper limb function, hand function, and combined upper limb and hand function, the kinematic feature sub-level of the function is determined based on the kinematic characteristics of the corresponding movement, the muscle activation timing coordination sub-level of the function is determined based on the value of the muscle activation timing coordination of the corresponding movement, and the abnormal compensatory activity sub-level of the function is determined based on the value of the abnormal compensatory activity of the corresponding movement. If two or more of the following are considered the lowest levels: kinematic feature sub-level, muscle activation timing coordination sub-level, and abnormal compensatory activity sub-level, the function level is determined to be the lowest level. If not, the abnormal compensatory activity sub-level is determined to be the lowest level. If so, a first level upper limit is determined, and the level that appears most frequently among the kinematic feature sub-level, muscle activation timing coordination sub-level, and abnormal compensatory activity sub-level is selected as a candidate level. The lower of the candidate level and the first level upper limit is determined as the function level. If not, a second level upper limit is determined, and the level that appears most frequently among the kinematic feature sub-level, muscle activation timing coordination sub-level, and abnormal compensatory activity sub-level is selected as a candidate level. The lower of the candidate level and the second level upper limit is determined as the function level.
8. A rehabilitation training program generation device based on multimodal data fusion, characterized in that, include: The data acquisition module is used to acquire video data and surface electromyography data of the target object during the process of the target object performing the evaluation action; The indicator acquisition module is used to acquire the kinematic characteristics of each movement in the upper limb movement, hand movement, and combined upper limb and hand movement based on the video data, and to acquire the muscle activation timing coordination and abnormal compensatory activity of the movement based on the surface electromyography data. The rating assessment module is used to assess the upper limb function level of the target object based on the kinematic characteristics of upper limb movements, muscle activation timing coordination, and abnormal compensatory activity; to assess the hand function level of the target object based on the kinematic characteristics of hand movements, muscle activation timing coordination, and abnormal compensatory activity; and to assess the overall upper limb and hand function level of the target object based on the kinematic characteristics of combined upper limb and hand movements, muscle activation timing coordination, and abnormal compensatory activity. The action recommendation module is used to match and recommend suitable training actions from the upper limb action sub-library, hand action sub-library, and upper limb and hand comprehensive action sub-library in the action library based on the target object's upper limb function level, hand function level, and upper limb and hand comprehensive function level.
9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the rehabilitation training program generation method based on multimodal data fusion as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the rehabilitation training program generation method based on multimodal data fusion as described in any one of claims 1 to 7.