Multi-scale Motion Rehabilitation Assessment Method, System, Computer Device and Storage Medium

Abstract

The present invention provides a multi-scale motion rehabilitation assessment method, system, computer device, and storage medium. The method includes the following steps: collecting limb rehabilitation exercise videos during the rehabilitation training of the object to be evaluated; for each frame image in the limb rehabilitation exercise videos, extracting feature information of the limb parts, including equivalent shape information, vector information, and joint point information; performing feature encoding on each limb part to generate weights corresponding to each limb part and action sequence encoding; based on the feature encoding results, constructing an assessment model, which is a sequential model for establishing the internal connection of actions, and outputting the assessment result of the rehabilitation training of the object to be evaluated. The present invention can comprehensively evaluate the training actions by integrating the body part information, limb vector features, and joint point features in the actions, so as to more accurately evaluate the training actions.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical equipment, and in particular to a multi-scale motion rehabilitation assessment method, system, computer equipment and storage medium. Background Art

[0002] An increasing number of patients with motor dysfunction require rehabilitation training to restore normal function, a long-term need. Furthermore, professional assessment methods are also required for individuals with normal exercise performance. Traditional training methods involve patients participating in a training program or rehabilitation session, supervised by a specialist, who prescribes one or more exercises. This can lead to rigid training regimens that lack flexibility, and patients may become overly reliant on expert guidance and supervision. Furthermore, this approach places significant strain on the healthcare system. Most importantly, without expert supervision, patients are unable to effectively manage their condition, resulting in a lack of timely understanding of their rehabilitation outcomes and significantly reduced compliance with training. Furthermore, when patients perform different types of rehabilitation exercises, variations in movement can lead to compensation using other body parts, which reflects differences in rehabilitation effectiveness and force application across different limbs. Therefore, an accurate and comprehensive rehabilitation assessment method is needed, particularly one tailored to the specific characteristics of individuals with limb dysfunction.

[0003] Patent (CN110941990A) describes a method for human motion assessment based on skeletal keypoints. This method assesses human motion based on calculated body posture angles. This method focuses on abnormal user posture, the lack of correlation analysis between various moving parts, the assessment of specific movement capabilities, and motor compensation issues caused by insufficient or irregular movement capabilities.

[0004] The patent (CN202211107295.3) introduces a multi-scale spatiotemporal decomposition network method for upper limb rehabilitation movement recognition. This method achieves improved recognition accuracy by randomly enhancing the sample video and background light. The main focus is on removing background interference in the use case and enhancing data processing.

[0005] However, rehabilitation movements differ significantly from common movement datasets: (1) Rehabilitation training movements are simpler and focus more on the motor abilities of certain parts (the hemiplegic side or the side with motor dysfunction); (2) The degree of movement completion and the degree of completion of related parts (such as motor compensation) are analyzed, that is, the correlation between the various moving parts when the user completes the movement. The above literature lacks evaluation methods in this regard. Summary of the Invention

[0006] The object of the present invention is to provide a rehabilitation training evaluation method for people with motor function disorders in view of the problems existing in the above-mentioned prior art. It focuses more on the recovery or enhancement of the motor ability of the affected limb, and its action characteristics are simple. During the movement rehabilitation process, it focuses more on evaluating the motor ability of one or several associated parts of this group. Especially when the user is evaluating the training of a specific part, if this part cannot complete the specified action and is completed by other parts instead (motor compensation), it is necessary to comprehensively evaluate the motor relevance of several parts.

[0007] The technical solution for achieving the object of the present invention is: a multi-scale movement rehabilitation evaluation method, and the method includes the following steps:

[0008] Step 1, collect the limb rehabilitation movement video during the rehabilitation training of the object to be evaluated;

[0009] Step 2, for each frame image in the limb rehabilitation movement video, extract the feature information of the limb part, including equivalent shape information, vector information, and joint point information;

[0010] Step 3, perform feature encoding on each limb part to generate the weight corresponding to each limb part and the action sequence encoding;

[0011] Step 4, based on the feature encoding result, construct an evaluation model, which is a time-sequential model establishing the internal connection of actions, and output the evaluation result of the rehabilitation training of the object to be evaluated.

[0012] Further, the limb part division in Step 2, as well as the equivalent shape information, vector information, and joint point information are specifically:

[0013] Right hip p1: equivalent shape rect1, vector v1, joint points (j0, j1);

[0014] Right thigh p2: equivalent shape rect2, vector v2, joint points (j1, j2);

[0015] Right calf p3: equivalent shape rect3, vector v3, joint points (j2, j3);

[0016] Left hip p4: equivalent shape rect4, vector v4, joint points (j0, j4);

[0017] Left thigh p5: equivalent shape rect5, vector v5, joint points (j4, j5);

[0018] Left calf p6: equivalent shape rect6, vector v6, joint points (j5, j6);

[0019] Torso p7: equivalent shape rect7, vector v7, joint points (j0, j7, j8);

[0020] Neck p8: equivalent shape rect8, vector v8, joint points (j8, j9, j 10 );

[0021] Left shoulder p9: equivalent shape rect9, vector v9, joint points (j8, j 11 );

[0022] Left posterior arm p 10 : equivalent shape rect 10 , vector v 10 , joint points (j 11 , j 12 );

[0023] Left forearm p 11 : equivalent shape rect 11 , vector v 11 , joint points (j 12 , j 13 );

[0024] Right shoulder p 12 : equivalent shape rect 12 , vector v 12 , joint points (j8, j 14 );

[0025] Right forearm p 13 : equivalent shape rect 13 , vector v 13 , joint points (j 14 , j 15 );

[0026] Right posterior arm p 14 : equivalent shape rect 14 , vector v 14 , joint points (j 15 , j 16 ).

[0027] Furthermore, the feature encoding described in step 3 includes:

[0028] Position encoding, used to generate weights corresponding to limb parts;

[0029] Sequence encoding, used to encode the rehabilitation training action sequence.

[0030] Furthermore, the position encoding specifically includes:

[0031] Obtain the features of each limb part through max pooling and average pooling operations respectively of The significant features and the mean features :

[0032]

[0033] In the formula, and respectively represent the max-pooling and mean-pooling operations, and N×M is the spatial dimension of the feature ; the feature includes features such as equivalent shape information, vector information, and joint point information;

[0034] Based on the significant features and the mean features , weights corresponding to each limb part are generated:

[0035]

[0036] In the formula, represents the weight corresponding to the limb part , represents concatenation, MLP represents the MLP encoding method / model, is the activation function, and n represents the total number of limb parts.

[0037] Furthermore, the sequence encoding specifically includes:

[0038] Construct the action set at time t in the limb rehabilitation exercise video :

[0039]

[0040] In the formula, represents the action signal of the limb part at time t, , represents a complete action cycle;

[0041] Construct the action set within a complete action cycle : :

[0042]

[0043] In the formula, ; represents the action set at time

[0044] Select the key actions related to action assessment from the action set to construct the key action set:

[0045]

[0046] In the formula, ; here, is not necessarily the sequential moment, corresponding to the moment where the key action for screening is located;

[0047] Perform action sequence encoding:

[0048]

[0049] In the formula, is a characteristic constant, determining the encoding broadening near the sequence, is a normalization constant.

[0050] Furthermore, the evaluation model described in step 4 includes a feature extraction module, a feature fusion module, a first mapping module, and a second mapping module;

[0051] The feature extraction module includes: a first feature extraction unit for respectively extracting the temporal features of equivalent shape information, vector information, and joint point information; a second feature extraction unit for respectively extracting the spatial features of equivalent shape information, vector information, and joint point information;

[0052] The feature fusion module includes: a first feature fusion unit for fusing the temporal features of equivalent shape information, vector information, and joint point information; a second feature fusion unit for fusing the spatial features of equivalent shape information, vector information, and joint point information;

[0053] The first mapping module is used to map the fused spatial features and temporal features;

[0054] The second mapping module is used to establish the relationship between the features mapped by the first mapping module and the evaluation result.

[0055] Furthermore, the second feature extraction unit respectively extracts the spatial features of equivalent shape information, vector information, and joint point information, specifically including:

[0056] (1) Extraction of spatial features of equivalent shape information

[0057] For each limb part , input the equivalent shape information of the object to be evaluated into the CNN model to extract the equivalent shape information features ; at the same time, input the equivalent shape information of the standard rehabilitation action into the CNN model to extract the standard equivalent shape information features ;

[0058] Calculate the limb part The corresponding equivalent shape information loss function :

[0059]

[0060] In the formula, is the activation function;

[0061] Based on the weights corresponding to each limb part obtained in step 3, obtain the spatial features of the equivalent shape information of the whole body :

[0062]

[0063] In the formula, is the activation function, represents the weight corresponding to the limb part n represents the total number of limb parts;

[0064] (2) Extraction of spatial features of vector information

[0065] For each limb part , input the vector information of the object to be evaluated into the CNN model to extract vector information features ; at the same time, input the vector information of the standard rehabilitation action into the CNN model to extract standard vector information features ;

[0066] Calculate the vector information loss function corresponding to the limb part : :

[0067]

[0068] Based on the weights corresponding to each limb part obtained in step 3, obtain the spatial features of the vector information of the whole body :

[0069]

[0070] (3) Extraction of spatial features of joint point information

[0071] For each limb part , input the joint point information of the object to be evaluated into the CNN model to extract joint point features ; at the same time, input the joint point information of the standard rehabilitation action into the CNN model to extract standard joint point information features ;

[0072] Calculate the joint point information loss function corresponding to the limb part : :

[0073]

[0074] Obtain the spatial features of the joint point information of the whole body based on the weights corresponding to each limb part obtained in step 3 :

[0075]

[0076] Furthermore, the first feature extraction unit extracts the temporal features of the equivalent shape information, vector information, and joint point information respectively, specifically including:

[0077] (1) At time t, input the equivalent shape information / vector information / joint point information of each limb part into the TCN module respectively:

[0078]

[0079] Among them, the TCN module includes a fully connected convolution for feature alignment , and a module that sets the time interval between multiple layers of inputs through the kernel function stride and convolution parameters; represents the equivalent shape information / vector information / joint point information of the limb part at time t ; represents the temporal features of the equivalent shape information / vector information / joint point information output by the TCN module;

[0080] Based on the weights corresponding to each limb part obtained in step 3, fuse the temporal features output by the equivalent shape information / vector information / joint point information of all limb parts at time t after passing through the TCN module to obtain :

[0081]

[0082] Let ;

[0083] (2) Based on the process in (1) above, obtain the fused feature set output by the TCN module within a complete action cycle T :

[0084]

[0085] In the formula, ;

[0086] (3) Calculate the attention degree of the key action time node coding at time t to time j:

[0087]

[0088] Among them,

[0089]

[0090] In the formula, represents the weight, which is used to represent the degree of attention to the key action time node encoding at time t for time node j,

[0091] is the key action sequence encoding at time node j calculated based on the method in step 3, and d is the dimension of;

[0092] (4) Use the weight and to update the fused feature set to obtain :

[0093]

[0094] Among them,

[0095]

[0096] Furthermore, the process of feature fusion by the first feature fusion unit / second feature fusion unit specifically includes:

[0097] After passing the temporal features / spatial features of the joint point information and vector information through the attention model ATT respectively, generate a new first feature sequence through addition operation;

[0098] After passing the temporal features / spatial features of the new first feature sequence and the equivalent shape information through the attention model ATT respectively, generate a new second feature sequence through addition operation.

[0099] Furthermore, the first mapping module maps the fused spatial features and temporal features, specifically: input the fused spatial features and temporal features into a fully connected layer to establish a feature mapping.

[0100] Compared with the prior art, the remarkable advantages of the present invention are:

[0101] (1) Multi-scale comprehensive evaluation: The present invention comprehensively considers the equivalent shape, vector, and joint point information of the limb parts. It comprehensively considers the morphology, movement direction, and joint movement of each part of the body, and accurately reflects the action completion quality and limb function status. For example, when evaluating the upper limb rehabilitation training of hemiplegic patients, not only the flexion and extension movements of the arm (reflected by vector information) are concerned, but also the change in joint point position (joint point information) and the contour of the arm muscles (equivalent shape information) are combined to comprehensively judge the training effect and avoid the one-sidedness of single-dimensional evaluation.

[0102] (2) Consider strong action relevance: For the characteristics of actions, pay attention to the movement relevance of each part and the movement compensation situation. Through feature encoding and evaluation models, analyze the interaction of different parts during action execution and accurately judge compensatory behaviors. For example, when a patient performs leg rehabilitation training, if excessive force in the hip compensates for the knee function, the present invention can accurately capture and analyze it.

[0103] (3) Evaluation method combining spatio-temporal features: The constructed evaluation model is a spatio-temporal feature model for establishing the internal connection of actions, which can effectively handle the changes of actions over time. By analyzing the features of each stage in the action sequence, dynamically evaluate the rehabilitation training effect.

[0104] The present invention will be further described in detail below with reference to the accompanying drawings. Description of the Drawings

[0105] Figure 1 It is a principle block diagram of the multi-scale motion rehabilitation evaluation method in an embodiment.

[0106] Figure 2 It is a schematic diagram of human key points in an embodiment.

[0107] Figure 3 It is a schematic diagram of spatial feature extraction in an embodiment.

[0108] Figure 4 It is a schematic diagram of time feature extraction in an embodiment.

[0109] Figure 5 It is a schematic diagram of feature fusion in an embodiment. Specific Embodiments

[0110] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0111] It should be noted that if there are directional indications (such as up, down, left, right, front, back...) involved in the embodiments of the present invention, the directional indications are only used to explain the relative positional relationship and movement situation between components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will also change accordingly.

[0112] In addition, if the descriptions such as "first" and "second" are involved in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and should not be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between various embodiments may be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0113] In one embodiment, in combination with Figure 1 , a multi-scale motion rehabilitation assessment method is provided, and the method includes the following steps:

[0114] Step 1, collect the limb rehabilitation exercise video during the rehabilitation training of the object to be evaluated;

[0115] Step 2, for each frame image in the limb rehabilitation exercise video, extract the feature information of the limb part, including equivalent shape information, vector information, and joint point information;

[0116] Step 3, perform feature encoding on each limb part to generate the corresponding weight and action sequence encoding for each limb part;

[0117] Step 4, based on the feature encoding result, construct an evaluation model, which is a temporal model that establishes the internal connection of actions, and output the evaluation result of the rehabilitation training of the object to be evaluated.

[0118] Here, the evaluation model can directly output the result, and the method of mapping and looking up tables is adopted, which will be introduced later.

[0119] If the evaluation model cannot directly output, the above process can be adjusted to:

[0120] Step 1, collect the limb rehabilitation exercise video during the rehabilitation training of the evaluated object and the limb rehabilitation exercise video during the rehabilitation training of the object to be evaluated;

[0121] Step 2, for each frame image in the limb rehabilitation exercise video, extract the feature information of the limb part, including equivalent shape information, vector information, and joint point information;

[0122] Step 3, perform feature encoding on each limb part to generate the corresponding weight and action sequence encoding for each limb part;

[0123] Step 4, based on the feature encoding result of the evaluated object, construct and train an evaluation model, which is a temporal model that establishes the internal connection of actions;

[0124] Step 5: Input the feature coding result of the object to be evaluated into the trained evaluation model, and output the evaluation result of the rehabilitation training of the object to be evaluated.

[0125] Further, in one embodiment, in combination with Figure 2 , the limb part division in step 2, and the equivalent shape information, vector information, and joint point information are specifically as follows:

[0126] Right hip p1: Equivalent shape rect1, vector v1, joint points (j0, j1);

[0127] Right thigh p2: Equivalent shape rect2, vector v2, joint points (j1, j2);

[0128] Right calf p3: Equivalent shape rect3, vector v3, joint points (j2, j3);

[0129] Left hip p4: Equivalent shape rect4, vector v4, joint points (j0, j4);

[0130] Left thigh p5: Equivalent shape rect5, vector v5, joint points (j4, j5);

[0131] Left calf p6: Equivalent shape rect6, vector v6, joint points (j5, j6);

[0132] Trunk p7: Equivalent shape rect7, vector v7, joint points (j0, j7, j8);

[0133] Neck p8: Equivalent shape rect8, vector v8, joint points (j8, j9, j 10 );

[0134] Left shoulder p9: Equivalent shape rect9, vector v9, joint points (j8, j 11 );

[0135] Left posterior arm p 10 : Equivalent shape rect 10 , vector v 10 , joint points (j 11 , j 12 );

[0136] Left forearm p 11 : Equivalent shape rect 11 , vector v 11 , joint points (j 12 , j 13 );

[0137] Right shoulder p 12 : Equivalent shape rect12 , vector v 12 , joint points (j8, j 14 );

[0138] Right forearm p 13 : equivalent shape rect 13 , vector v 13 , joint points (j 14 , j 15 );

[0139] Right hindarm p 14 : equivalent shape rect 14 , vector v 14 , joint points (j 15 , j 16 ).

[0140] Here, the vector information reflects the flexion and extension movements of various parts such as the arm, the joint point information reflects the position changes of the joint points, and the equivalent shape information reflects the muscle contours of various parts such as the arm.

[0141] Furthermore, in one embodiment, the feature encoding in step 3 includes:

[0142] Position encoding, used to generate weights corresponding to limb parts;

[0143] Sequence encoding, used to encode the rehabilitation training action sequence.

[0144] Here, preferably, in some embodiments, the position encoding specifically includes:

[0145] Obtaining the significant features of each limb part and the mean features through max pooling and average pooling operations respectively :

[0146]

[0147] ,

[0148] wherein, 、 respectively represent max pooling and average pooling operations, N×M is the spatial dimension of the feature ; the feature includes features of equivalent shape information, vector information and joint point information;

[0149] Based on the significant features and the mean features , generate each limb part Corresponding weight:

[0150]

[0151] In the formula, represents the limb part corresponding weight, represents splicing, and MLP represents the MLP encoding method / model, is the activation function, and n represents the total number of limb parts.

[0152] Here, preferably, in some embodiments, the sequence encoding specifically includes:

[0153] Construct the action set at time t in the limb rehabilitation exercise video :

[0154]

[0155] In the formula, represents the action signal of the limb part at time t, , represents a complete action cycle;

[0156] Construct a complete action cycle within the action set :

[0157]

[0158] In the formula, ; represents the action set at time

[0159] Select the key actions related to action assessment from the action set to construct the key action set:

[0160]

[0161] In the formula, ; here, is not necessarily the sequential time, corresponding to the time when the selected key action is located;

[0162] Perform action sequence encoding:

[0163]

[0164] In the formula, is the characteristic constant, which determines the encoding broadening near the sequence, is the normalization constant.

[0165] Further, in one of the embodiments, the evaluation model in step 4 includes a feature extraction module, a feature fusion module, a first mapping module, and a second mapping module;

[0166] The feature extraction module includes: a first feature extraction unit for respectively extracting the temporal features of the equivalent shape information, vector information, and joint point information; a second feature extraction unit for respectively extracting the spatial features of the equivalent shape information, vector information, and joint point information;

[0167] The feature fusion module includes: a first feature fusion unit for fusing the temporal features of the equivalent shape information, vector information, and joint point information; a second feature fusion unit for fusing the spatial features of the equivalent shape information, vector information, and joint point information;

[0168] The first mapping module is used to map the fused spatial features and temporal features;

[0169] The second mapping module is used to establish the relationship between the features mapped by the first mapping module and the evaluation results.

[0170] Here, this relationship is established by means of a look-up table (a pre-established relationship table between features and evaluation results), so the evaluation results can be directly output.

[0171] Preferably, in some embodiments, in combination with Figure 3 , the second feature extraction unit respectively extracts the spatial features of the equivalent shape information, vector information, and joint point information, specifically including:

[0172] (1) Extraction of spatial features of equivalent shape information

[0173] For each limb part , the equivalent shape information of the object to be evaluated is input into the CNN model to extract the equivalent shape information features ; at the same time, the equivalent shape information of the standard rehabilitation action is input into the CNN model to extract the standard equivalent shape information features ;

[0174] Calculate the equivalent shape information loss function corresponding to the limb part : :

[0175]

[0176] In the formula, is the activation function;

[0177] Based on the weights corresponding to each limb part obtained in step 3, obtain the spatial features of the equivalent shape information of the whole body :

[0178]

[0179] In the formula, is the activation function, represents the limb part corresponding weight, and n represents the total number of limb parts;

[0180] (2) Spatial feature extraction of vector information

[0181] For each limb part , input the vector information of the object to be evaluated into the CNN model to extract the vector information features ; at the same time, input the vector information of the standard rehabilitation action into the CNN model to extract the standard vector information features ;

[0182] Calculate the vector information loss function corresponding to the limb part : :

[0183]

[0184] Based on the weights corresponding to each limb part obtained in step 3, obtain the spatial features of the vector information of the whole body :

[0185]

[0186] (3) Spatial feature extraction of joint point information

[0187] For each limb part , input the joint point information of the object to be evaluated into the CNN model to extract the joint point features ; at the same time, input the joint point information of the standard rehabilitation action into the CNN model to extract the standard joint point information features ;

[0188] Calculate the joint point information loss function corresponding to the limb part : :

[0189]

[0190] Based on the weights corresponding to each limb part obtained in step 3, obtain the spatial features of the joint point information of the whole body :

[0191]

[0192] Preferably, in some embodiments, combineFigure 4 The first feature extraction unit extracts the temporal features of the equivalent shape information, vector information, and joint point information respectively, specifically including:

[0193] (1) At time t, input the equivalent shape information / vector information / joint point information of each limb part into the TCN module respectively:

[0194]

[0195] Among them, the TCN module includes a fully connected convolution for feature alignment and a module that sets the time interval between multiple layers of inputs through the kernel function stride and convolution parameters; represents the equivalent shape information / vector information / joint point information of the limb part at time t ; represents the temporal feature of the equivalent shape information / vector information / joint point information output by the TCN module;

[0196] Based on the weights corresponding to each limb part obtained in step 3, fuse the temporal features output by the equivalent shape information / vector information / joint point information of all limb parts at time t after passing through the TCN module to obtain :

[0197]

[0198] Let ;

[0199] (2) Based on the process in (1) above, obtain the fused feature set output by the TCN module within a complete action cycle T:

[0200]

[0201] In the formula, ;

[0202] (3) Calculate the attention degree of the key action time node encoding at time t to that at time j:

[0203]

[0204] Among them,

[0205]

[0206] In the formula, represents the weight, which is used to represent the attention degree of the key action time node encoding at time t to that at time j,

[0207] The key action sequence encoding at time j calculated based on the method in step 3, where d is the dimension of

[0208] (4) Using the weights and to update the fused feature set to obtain :

[0209]

[0210] where

[0211]

[0212] Preferably, in some embodiments, in combination with Figure 5 , the process of feature fusion by the first feature fusion unit / second feature fusion unit specifically includes:

[0213] After passing the temporal features / spatial features of the joint point information and vector information through the attention model ATT respectively, generating a new first feature sequence through addition operation;

[0214] After passing the temporal features / spatial features of the new first feature sequence and the equivalent shape information through the attention model ATT respectively, generating a new second feature sequence through addition operation.

[0215] where the operation of the attention model ATT is specifically:

[0216]

[0217]

[0218] In the formula, , are the feature sequences of the corresponding channels respectively, , , , is the learned projection matrix, is the dimension of the key-value set, Q and K are the learned query and key vectors, is the tensor dimension space of the corresponding input feature channel.

[0219] Preferably, in some embodiments, the first mapping module maps the fused spatial features and temporal features, specifically: inputting the fused spatial features and temporal features into a fully connected layer to establish a feature mapping.

[0220] In one embodiment, a multi-scale motion rehabilitation evaluation system is provided, and the system includes:

[0221] The first module is used to collect limb rehabilitation exercise videos during the rehabilitation training of the object to be evaluated;

[0222] The second module is used to extract the feature information of the limb parts for each frame image in the limb rehabilitation exercise video, including equivalent shape information, vector information, and joint point information;

[0223] The third module is used to perform feature encoding on each limb part to generate the weights corresponding to each limb part and the action sequence encoding;

[0224] The fourth module is used to construct an evaluation model based on the feature encoding result. This model is a sequential model that establishes the internal connection of actions and outputs the evaluation result of the rehabilitation training of the object to be evaluated.

[0225] For the specific limitations of the multi-scale motion rehabilitation evaluation system, reference can be made to the limitations of the multi-scale motion rehabilitation evaluation method in the above text, which will not be elaborated here. Each module in the above multi-scale motion rehabilitation evaluation system can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in the form of hardware or independent of it, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0226] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following is implemented:

[0227] Step 1: Collect limb rehabilitation exercise videos during the rehabilitation training of the object to be evaluated;

[0228] Step 2: For each frame image in the limb rehabilitation exercise video, extract the feature information of the limb parts, including equivalent shape information, vector information, and joint point information;

[0229] Step 3: Perform feature encoding on each limb part to generate the weights corresponding to each limb part and the action sequence encoding;

[0230] Step 4: Based on the feature encoding result, construct an evaluation model. This model is a sequential model that establishes the internal connection of actions and outputs the evaluation result of the rehabilitation training of the object to be evaluated.

[0231] For the specific limitations of each step, reference can be made to the limitations of the multi-scale motion rehabilitation evaluation method in the above text, which will not be elaborated here.

[0232] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following is implemented:

[0233] Step 1: Collect the limb rehabilitation exercise videos during the rehabilitation training of the object to be evaluated.

[0234] Step 2: For each frame image in the limb rehabilitation exercise video, extract the feature information of the limb parts, including equivalent shape information, vector information, and joint point information.

[0235] Step 3: Perform feature encoding on each limb part to generate the weights corresponding to each limb part and the action sequence encoding.

[0236] Step 4: Based on the feature encoding results, construct an evaluation model, which is a sequential model that establishes the internal connection of actions, and output the evaluation result of the rehabilitation training of the object to be evaluated.

[0237] For the specific limitations of each step, reference can be made to the limitations on the multi-scale motion rehabilitation evaluation method in the above text, which will not be elaborated here.

[0238] It should be noted that the CNN model and the like adopted in the present invention are preferably, but not limited to these models, and the application of other models also falls within the protection scope of the present invention.

[0239] The above has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification only illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-scale motion rehabilitation assessment method, characterized in that, The method includes the following steps: Step 1, collect the limb rehabilitation exercise videos during the rehabilitation training of the object to be evaluated; Step 2, for each frame image in the limb rehabilitation exercise video, extract the feature information of the limb parts, including equivalent shape information, vector information, and joint point information; Step 3, perform feature encoding on each limb part to generate the weights corresponding to each limb part and the action sequence encoding; Step 4, based on the feature encoding results, construct an evaluation model, which is a sequential model that establishes the internal connection of actions, and output the evaluation result of the rehabilitation training of the object to be evaluated; The feature encoding includes: Position encoding, used to generate the weights corresponding to the limb parts; Sequence encoding, used to perform the encoding of the rehabilitation training action sequence; The sequence encoding specifically includes: Construct the set of actions at time t in the limb rehabilitation exercise video : ； In the formula, represents the motion signal of the limb part at time t , , represents a complete motion cycle; Construct a complete action cycle The set of actions within : ； In the formula, ; represents the set of actions at the moment; Screen key actions related to action evaluation from the set of actions to construct a set of key actions: ； In the formula, ; Here, is not necessarily the sequential moment and corresponds to the moment when the key action to be screened is located; Perform action sequence encoding: ； In the formula, is a characteristic constant that determines the coding broadening near the sequence, and is a normalization constant.

2. The multi-scale motion rehabilitation assessment method according to claim 1, wherein, The limb part division in Step 2, and the equivalent shape information, vector information, and joint point information are specifically: Right hip p1: equivalent shape rect1, vector v1, joint points (j0, j1); Right thigh p2: equivalent shape rect2, vector v2, joint points (j1, j2); Right calf p3: equivalent shape rect3, vector v3, joint points (j2, j3); Left hip p4: equivalent shape rect4, vector v4, joint points (j0, j4); Left thigh p5: equivalent shape rect5, vector v5, joint points (j4, j5); Left calf p6: equivalent shape rect6, vector v6, joint points (j5, j6); Trunk p7: equivalent shape rect7, vector v7, joint points (j0, j7, j8); Neck p8: Equivalent shape rect8, vector v8, joint points (j8, j9, j 10 ); Left shoulder p9: Equivalent shape rect9, vector v9, joint points (j8, j 11 ); Left posterior arm p 10 : Equivalent shape rect 10 , vector v 10 , joint point (j 11 , j 12 ); Left forearm p 11 : Equivalent shape rect 11 , vector v 11 , joint point (j 12 , j 13 ); Right shoulder p 12 : Equivalent shape rect 12 , vector v 12 , joint points (j8, j 14 ); Right forearm p 13 : Equivalent shape rect 13 , vector v 13 , joint point (j 14 , j 15 ); Right posterior arm p 14 : Equivalent shape rect 14 , vector v 14 , joint point (j 15 , j 16 ).

3. The multi-scale motion rehabilitation assessment method according to claim 1, characterized in that The position encoding specifically includes: Obtain the features of each limb part through max pooling and average pooling operations respectively of the significant features and the average features : ； , ； In the formula, and represent the max pooling and average pooling operations respectively, and N×M is the spatial dimension of the feature ; the feature is a feature including equivalent shape information, vector information and joint point information; Based on the said significant features and the mean features , generate the weights corresponding to each limb part : ； In the formula, represents the limb part corresponding weight, represents splicing, MLP represents the MLP encoding method / model, is the activation function, and n represents the total number of limb parts.

4. The multi-scale motion rehabilitation assessment method according to claim 1, characterized in that The evaluation model in Step 4 includes a feature extraction module, a feature fusion module, a first mapping module, and a second mapping module; The feature extraction module includes: a first feature extraction unit, used to extract the temporal features of the equivalent shape information, vector information, and joint point information respectively; a second feature extraction unit, used to extract the spatial features of the equivalent shape information, vector information, and joint point information respectively; The feature fusion module includes: a first feature fusion unit, used to fuse the temporal features of the equivalent shape information, vector information, and joint point information; a second feature fusion unit, used to fuse the spatial features of the equivalent shape information, vector information, and joint point information; The first mapping module is used to map the fused spatial features and temporal features; The second mapping module is used to establish the relationship between the features mapped by the first mapping module and the evaluation result.

5. The multi-scale motion rehabilitation assessment method according to claim 4, characterized in that, The second feature extraction unit extracts the spatial features of the equivalent shape information, vector information, and joint point information respectively, specifically including: (1) Extraction of the spatial features of the equivalent shape information For each limb part , input the equivalent shape information of the object to be evaluated into the CNN model to extract the equivalent shape information features ; at the same time, input the equivalent shape information of the standard rehabilitation movement into the CNN model to extract the standard equivalent shape information features ; Calculating limb parts Corresponding equivalent shape information loss function : ； In the formula, is the activation function; Based on the weights corresponding to each limb part obtained in step 3, obtain the spatial features of the equivalent shape information of the whole body : ； In the formula, is the activation function, represents the limb part corresponding weight, and n represents the total number of limb parts; (2) Extraction of the spatial features of the vector information For each body part , input the vector information of the object to be evaluated into the CNN model to extract the vector information features ; at the same time, input the vector information of the standard rehabilitation movement into the CNN model to extract the standard vector information features ; Calculating limb parts Corresponding vector information loss function : ； Based on the weights corresponding to each limb part obtained in step 3, obtain the spatial features of the vector information of the whole body : ； (3) Extraction of the spatial features of the joint point information For each body part , input the joint point information of the object to be evaluated into the CNN model to extract joint point features ; at the same time, input the joint point information of the standard rehabilitation movement into the CNN model to extract the standard joint point information features ; Calculating limb parts Corresponding joint point information loss function : ； Based on the weights corresponding to each limb part obtained in step 3, obtain the spatial features of the joint point information of the whole body : 。 6. The multi-scale motion rehabilitation assessment method according to claim 4, wherein The first feature extraction unit extracts the temporal features of the equivalent shape information, vector information, and joint point information respectively, specifically including: At time t, the equivalent shape information / vector information / joint point information of each limb part is respectively input into the TCN module: ； Among them, the TCN module includes a fully connected convolution for feature alignment , and a module that sets the time interval between multiple layers of inputs through kernel function strides and convolution parameters ; represents the equivalent shape information / vector information / joint point information of the limb part at time t ; represents the time features of the equivalent shape information / vector information / joint point information output by the TCN module Based on the weights corresponding to each limb part obtained in step 3, fuse the temporal features output by the TCN module for the equivalent shape information / vector information / joint point information of all limb parts at time t to obtain :[[]]END]] ； Let ; (2) Obtain the set of fused features output by the TCN module within a complete action cycle T based on the process in (1) above : ； In the formula, ; (3) Calculate the attention degree of the key action time node encoding at time t to time j: ； Where, ； In the formula, represents the weight, which is used to indicate the degree of attention to the key action time node encoding at the j-th moment at the t-th moment. The key action sequence encoding at time j calculated based on the method in step 3, where d is the dimension of (4) Using the said weights and update the fused feature set to obtain : ； Where, 。 7. The multi-scale motion rehabilitation assessment method according to claim 4, wherein The process of feature fusion by the first feature fusion unit / second feature fusion unit specifically includes: After passing the temporal features / spatial features of the joint point information and vector information through the attention model ATT respectively, a new first feature sequence is generated through addition operation; After passing the temporal features / spatial features of the new first feature sequence and the equivalent shape information through the attention model ATT respectively, a new second feature sequence is generated through addition operation.

8. The multi-scale motion rehabilitation assessment method according to claim 4, wherein, The first mapping module maps the fused spatial features and temporal features, specifically: inputting the fused spatial features and temporal features into a fully connected layer to establish a feature mapping.

9. A multi-scale motion rehabilitation evaluation system based on the method according to any one of claims 1 to 8, the system comprising: A first module for collecting limb rehabilitation exercise videos during the rehabilitation training of the object to be evaluated; A second module for extracting feature information of limb parts for each frame image in the limb rehabilitation exercise video, including equivalent shape information, vector information, and joint point information; A third module for performing feature encoding on each limb part to generate weights corresponding to each limb part and an action sequence encoding; A fourth module for constructing an evaluation model based on the feature encoding result, the model being a sequential model that establishes the internal connection of actions, and outputting an evaluation result for the rehabilitation training of the object to be evaluated.

10. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images