Multi-scale exercise rehabilitation evaluation method and system, computer equipment and storage medium

Abstract

The invention provides a multi-scale exercise rehabilitation assessment method and system, computer equipment and a storage medium. The method comprises the following steps: acquiring a limb rehabilitation exercise video of a to-be-assessed object in a rehabilitation training process; for each frame of image in the limb rehabilitation exercise video, feature information, including equivalent shape information, vector information and joint point information, of a limb part is extracted; performing feature coding on each limb part to generate a weight and an action sequence code corresponding to each limb part; based on the feature coding result, an evaluation model is constructed, the model is a time sequence model for establishing the internal relation of actions, and an evaluation result of rehabilitation training of the to-be-evaluated object is output. According to the method, the body part information, the limb vector features and the joint point features in the actions can be synthesized to comprehensively evaluate the training actions, so that the training actions are evaluated more accurately.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical equipment, and in particular to a multi-scale motion rehabilitation assessment method, system, computer equipment and storage medium. Background Art

[0002] More and more patients with motor dysfunction need rehabilitation training to restore normal body functions. This rehabilitation demand is long-term. At the same time, for ordinary people, whether the exercise movements are standard also requires professional evaluation methods. The traditional training method is to let users participate in training plans or rehabilitation courses and supervised by experts to arrange one or more exercises for users. This leads to the fact that the training program may be too rigid and lack flexibility, and patients may rely too much on expert guidance and supervision. In addition, this method puts great pressure on the healthcare system. The most important thing is that when users are separated from expert supervision, they cannot effectively manage their own conditions, so that they cannot understand their own rehabilitation effects in time, and the compliance of training will be greatly reduced. In addition, when patients perform different types of rehabilitation movements, due to differences in movements, they use other parts for movement compensation during the execution of the movements, which reflects the differences in rehabilitation effects and force points between different parts of the patient's limbs. Therefore, an accurate and comprehensive rehabilitation evaluation method is needed, especially an evaluation method for the characteristics of people with limb dysfunction.

[0003] The patent (CN110941990A) introduces a method for evaluating human motion based on skeletal key points. This method evaluates human motion based on the calculated human posture azimuth. This method mainly focuses on the abnormal posture of the user, the lack of correlation analysis of various motion parts, the evaluation method of the motion ability of specific parts, and the motion compensation problems caused by insufficient or irregular motion ability.

[0004] The patent (CN202211107295.3) introduces a multi-scale spatiotemporal decomposition network upper limb rehabilitation action recognition method, which realizes rehabilitation action recognition by randomly enhancing the collected sample video and enhancing the background light, thereby improving the recognition accuracy. It mainly focuses on removing background interference in the usage scene and enhancing data processing.

[0005] However, rehabilitation movements are significantly different from common movement datasets: (1) Rehabilitation training movements are simpler and focus more on the motor ability of certain parts (the hemiplegic side or the side with motor dysfunction); (2) The degree of completion of the movement and the degree of completion of the associated parts (such as motor compensation), that is, the correlation analysis of the various movement parts when the user completes the movement. The above literature lacks evaluation methods in this regard. Summary of the invention

[0006] The purpose of the present invention is to provide a rehabilitation training evaluation method for people with motor dysfunction in view of the problems existing in the above-mentioned prior art. It focuses more on the recovery or enhancement of the motor ability of the affected limb, and its action characteristics are simple movements. During the exercise rehabilitation process, it focuses more on evaluating the motor ability of a single or related part of this group. In particular, when the user is evaluating the training of a specific part, if the part cannot complete the specified action, and other parts replace it (motor compensation), it is necessary to comprehensively evaluate the motor correlation of several parts.

[0007] The technical solution to achieve the purpose of the present invention is: a multi-scale motor rehabilitation assessment method, the method comprising the following steps:

[0008] Step 1, collecting limb rehabilitation exercise videos of the subject to be evaluated during rehabilitation training;

[0009] Step 2, for each frame of the limb rehabilitation exercise video, extracting feature information of the limb parts, including equivalent shape information, vector information and joint point information;

[0010] Step 3, feature encoding of each limb part, generating weights and action sequence encoding corresponding to each limb part;

[0011] Step 4: construct an evaluation model based on the feature encoding result, which is a temporal model that establishes the internal connection of actions and outputs the evaluation result of the rehabilitation training of the object to be evaluated.

[0012] Furthermore, the limb part division in step 2, as well as the equivalent shape information, vector information and joint point information are specifically as follows:

[0013] Right hip p1: equivalent shape rect1, vector v1, joint point (j0, j1);

[0014] Right thigh p2: equivalent shape rect2, vector v2, joint point (j1, j2);

[0015] Right calf p3: equivalent shape rect3, vector v3, joint point (j2, j3);

[0016] Left hip p4: equivalent shape rect4, vector v4, joint point (j0, j4);

[0017] Left thigh p5: equivalent shape rect5, vector v5, joint point (j4, j5);

[0018] Left calf p6: equivalent shape rect6, vector v6, joint point (j5, j6);

[0019] Torso p7: equivalent shape rect7, vector v7, joint points (j0, j7, j8);

[0020] Neck p8: equivalent shape rect8, vector v8, joint points (j8, j9, j 10 );

[0021] Left shoulder p9: equivalent shape rect9, vector v9, joint point (j8, j 11 );

[0022] Left rear arm 10 : Equivalent shape rect 10 , vector v 10 、Joint point(j 11 , j 12 );

[0023] Left forearm 11 : Equivalent shape rect 11 , vector v 11 、Joint point(j 12 , j 13 );

[0024] Right shoulder 12 : Equivalent shape rect 12 , vector v 12 , joint point (j8, j 14 );

[0025] Right forearm 13 : Equivalent shape rect 13 , vector v 13 、Joint point(j 14 , j 15 );

[0026] Right rear arm 14 : Equivalent shape rect 14 , vector v 14 、Joint point(j 15 , j 16 ).

[0027] Furthermore, the feature coding in step 3 includes:

[0028] Position encoding, used to generate weights corresponding to limb parts;

[0029] Sequence coding is used for rehabilitation training action sequence coding.

[0030] Furthermore, the position coding specifically includes:

[0031] Each limb part is obtained by maximum pooling and mean pooling operations respectively. Features The salient features and mean characteristics :

[0032]

[0033] In the formula, , Represents the maximum pooling and mean pooling operations respectively, and N×M is the feature The spatial dimension of Features including equivalent shape information, vector information, and joint point information;

[0034] Based on the salient features and mean characteristics , generating each limb part The corresponding weights are:

[0035]

[0036] In the formula, Indicates body parts The corresponding weight, represents concatenation, MLP represents the MLP encoding method / model, is the activation function, and n represents the total number of limb parts.

[0037] Furthermore, the sequence encoding specifically includes:

[0038] Constructing the action set at time t in the limb rehabilitation exercise video :

[0039]

[0040] In the formula, Indicates the limb part at time t The action signal, , Represents a complete action cycle;

[0041] Constructing a complete action cycle Action Set :

[0042]

[0043] In the formula, ; express The action set of the moment;

[0044] From the action set Filter the key actions related to action evaluation and build a key action set:

[0045]

[0046] In the formula, ;here, It is not necessarily a sequential moment, but corresponds to the moment of the key action of screening;

[0047] Encode the action sequence:

[0048]

[0049] In the formula, is the characteristic constant, determined in The coding width near the sequence is widened. is the normalization constant.

[0050] Furthermore, the evaluation model in step 4 includes a feature extraction module, a feature fusion module, a first mapping module and a second mapping module;

[0051] The feature extraction module includes: a first feature extraction unit for respectively extracting time features of equivalent shape information, vector information and joint point information; a second feature extraction unit for respectively extracting space features of equivalent shape information, vector information and joint point information;

[0052] The feature fusion module includes: a first feature fusion unit for fusing the time features of equivalent shape information, vector information and joint point information; a second feature fusion unit for fusing the spatial features of equivalent shape information, vector information and joint point information;

[0053] The first mapping module is used to map the fused spatial features and temporal features;

[0054] The second mapping module is used to establish a relationship between the features mapped by the first mapping module and the evaluation results.

[0055] Furthermore, the second feature extraction unit extracts the spatial features of equivalent shape information, vector information and joint point information respectively, specifically including:

[0056] (1) Spatial feature extraction of equivalent shape information

[0057] For every body part , input the equivalent shape information of the object to be evaluated into the CNN model, and extract the equivalent shape information features At the same time, the equivalent shape information of the standard rehabilitation action is input into the CNN model to extract the standard equivalent shape information features ;

[0058] Counting limb parts The corresponding equivalent shape information loss function :

[0059]

[0060] In the formula, is the activation function;

[0061] Based on the weights corresponding to each limb part obtained in step 3, the spatial features of the equivalent shape information of the whole body are obtained. :

[0062]

[0063] In the formula, is the activation function, Indicates body parts The corresponding weights, n represents the total number of limb parts;

[0064] (2) Spatial feature extraction of vector information

[0065] For every body part , input the vector information of the object to be evaluated into the CNN model and extract the vector information features At the same time, the vector information of the standard rehabilitation action is input into the CNN model to extract the standard vector information features ;

[0066] Counting limb parts The corresponding vector information loss function :

[0067]

[0068] Based on the weights corresponding to each limb part obtained in step 3, the spatial characteristics of the vector information of the whole body are obtained. :

[0069]

[0070] (3) Spatial feature extraction of joint point information

[0071] For every body part , input the joint point information of the object to be evaluated into the CNN model and extract the joint point features At the same time, the joint point information of the standard rehabilitation action is input into the CNN model to extract the standard joint point information features ;

[0072] Counting limb parts Corresponding joint point information loss function :

[0073]

[0074] Based on the weights corresponding to each limb part obtained in step 3, the spatial features of the joint point information of the whole body are obtained. :

[0075]

[0076] Furthermore, the first feature extraction unit extracts time features of equivalent shape information, vector information and joint point information respectively, specifically including:

[0077] (1) At time t, move each limb part The equivalent shape information / vector information / joint point information are input into the TCN module respectively:

[0078]

[0079] Among them, the TCN module includes a fully connected convolution for feature alignment , and the time interval between multiple layers of input is set by the kernel function step size and convolution parameters Modules; Indicates the limb part at time t Equivalent shape information / vector information / joint point information; Represents the temporal characteristics of the equivalent shape information / vector information / joint point information output by the TCN module;

[0080] Based on the weights corresponding to each limb part obtained in step 3, the time features of the equivalent shape information / vector information / joint point information of all limb parts at time t output after the TCN module are fused to obtain :

[0081]

[0082] make ;

[0083] (2) Based on the process of (1) above, we obtain the fusion feature set output by the TCN module within a complete action cycle T. :

[0084]

[0085] In the formula, ;

[0086] (3) Calculate the degree of attention paid to the key action time node encoding at time j at time t:

[0087]

[0088] in,

[0089]

[0090] In the formula, represents the weight, which is used to indicate the degree of attention paid to the key action time node encoding at time j at time t.

[0091] is the key action sequence encoding at time j calculated based on step 3, and d is Dimensions;

[0092] (4) Using the weights and Update fusion feature set get :

[0093]

[0094] in,

[0095]

[0096] Furthermore, the process of the first feature fusion unit / the second feature fusion unit performing feature fusion specifically includes:

[0097] After the temporal features / spatial features of the joint point information and vector information are respectively passed through the attention model ATT, a new first feature sequence is generated through addition operation;

[0098] The new first feature sequence and the temporal features / spatial features of the equivalent shape information are respectively subjected to the attention model ATT, and then a new second feature sequence is generated through an addition operation.

[0099] Furthermore, the first mapping module maps the fused spatial features and temporal features, specifically by inputting the fused spatial features and temporal features into a fully connected layer to establish a feature mapping.

[0100] Compared with the prior art, the present invention has the following significant advantages:

[0101] (1) Multi-scale comprehensive evaluation: The present invention integrates the equivalent shape, vector and joint point information of limb parts. It comprehensively considers the morphology, movement direction and joint activity of various parts of the body, and accurately reflects the quality of movement completion and limb functional status. For example, when evaluating the upper limb rehabilitation training of hemiplegic patients, it not only pays attention to the arm flexion and extension movement (reflected by vector information), but also combines the changes in joint point position (joint point information) and arm muscle contour (equivalent shape information) to comprehensively judge the training effect and avoid the one-sidedness of single-dimensional evaluation.

[0102] (2) Consider the strong correlation of actions: Based on the characteristics of the action, pay attention to the correlation of movements of various parts and the compensation of movements. Through feature coding and evaluation models, analyze the interaction between different parts during the execution of the action and accurately judge the compensatory behavior. For example, if the patient undergoes leg rehabilitation training and the hip overexerts to compensate for the knee function, the present invention can accurately capture and analyze it.

[0103] (3) Evaluation method combining spatiotemporal features: The evaluation model constructed is a spatiotemporal feature model that establishes the internal connection of movements, which can effectively handle the changes of movements over time. By analyzing the characteristics of each stage in the movement sequence, the rehabilitation training effect can be dynamically evaluated.

[0104] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 It is a principle block diagram of a multi-scale motion rehabilitation assessment method in one embodiment.

[0106] Figure 2 Schematic diagram of key points of a human body in one embodiment.

[0107] Figure 3 It is a schematic diagram of spatial feature extraction in one embodiment.

[0108] Figure 4 A schematic diagram of time feature extraction in an embodiment.

[0109] Figure 5 Schematic diagram of feature fusion in an embodiment. DETAILED DESCRIPTION

[0110] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0111] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0112] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0113] In one embodiment, in combination Figure 1 , provides a multi-scale motor rehabilitation assessment method, the method comprising the following steps:

[0114] Step 1, collecting limb rehabilitation exercise videos of the subject to be evaluated during rehabilitation training;

[0115] Step 2, for each frame of the limb rehabilitation exercise video, extracting feature information of the limb parts, including equivalent shape information, vector information and joint point information;

[0116] Step 3, feature encoding of each limb part, generating weights and action sequence encoding corresponding to each limb part;

[0117] Step 4: construct an evaluation model based on the feature encoding result, which is a temporal model that establishes the internal connection of actions and outputs the evaluation result of the rehabilitation training of the object to be evaluated.

[0118] Here, the evaluation model can directly output the results, using a mapping and table lookup method, which will be introduced later.

[0119] If the evaluation model cannot be output directly, the above process can be adjusted as follows:

[0120] Step 1, collecting limb rehabilitation exercise videos of the evaluated subject during rehabilitation training, and collecting limb rehabilitation exercise videos of the subject to be evaluated during rehabilitation training;

[0121] Step 2, for each frame of the limb rehabilitation exercise video, extracting feature information of the limb parts, including equivalent shape information, vector information and joint point information;

[0122] Step 3, feature encoding of each limb part, generating weights and action sequence encoding corresponding to each limb part;

[0123] Step 4, based on the feature encoding results of the evaluated object, construct and train an evaluation model, which is a temporal model for establishing the internal connection of actions;

[0124] Step 5: input the feature coding result of the object to be evaluated into the trained evaluation model, and output the evaluation result of the rehabilitation training of the object to be evaluated.

[0125] Further, in one embodiment, in combination Figure 2 , the limb part division in step 2, as well as the equivalent shape information, vector information and joint point information are specifically as follows:

[0126] Right hip p1: equivalent shape rect1, vector v1, joint point (j0, j1);

[0127] Right thigh p2: equivalent shape rect2, vector v2, joint point (j1, j2);

[0128] Right calf p3: equivalent shape rect3, vector v3, joint point (j2, j3);

[0129] Left hip p4: equivalent shape rect4, vector v4, joint point (j0, j4);

[0130] Left thigh p5: equivalent shape rect5, vector v5, joint point (j4, j5);

[0131] Left calf p6: equivalent shape rect6, vector v6, joint point (j5, j6);

[0132] Torso p7: equivalent shape rect7, vector v7, joint points (j0, j7, j8);

[0133] Neck p8: equivalent shape rect8, vector v8, joint points (j8, j9, j 10 );

[0134] Left shoulder p9: equivalent shape rect9, vector v9, joint point (j8, j 11 );

[0135] Left rear arm 10 : Equivalent shape rect 10 , vector v 10 、Joint point(j 11 , j 12 );

[0136] Left forearm 11 : Equivalent shape rect 11 , vector v 11 、Joint point(j 12 , j 13 );

[0137] Right shoulder 12 : Equivalent shape rect12 , vector v 12 , joint point (j8, j 14 );

[0138] Right forearm 13 : Equivalent shape rect 13 , vector v 13 、Joint point(j 14 , j 15 );

[0139] Right rear arm 14 : Equivalent shape rect 14 , vector v 14 、Joint point(j 15 , j 16 ).

[0140] Here, the vector information reflects the flexion and extension movements of the arms and other parts of the body, the joint point information reflects the changes in the joint point positions, and the equivalent shape information reflects the muscle contours of the arms and other parts of the body.

[0141] Further, in one embodiment, the feature coding in step 3 includes:

[0142] Position encoding, used to generate weights corresponding to limb parts;

[0143] Sequence coding is used for rehabilitation training action sequence coding.

[0144] Preferably, in some embodiments, the position coding specifically includes:

[0145] Each limb part is obtained by maximum pooling and mean pooling operations respectively. Features The salient features and mean characteristics :

[0146]

[0147] ,

[0148] In the formula, , Represents the maximum pooling and mean pooling operations respectively, and N×M is the feature The spatial dimension of Features including equivalent shape information, vector information, and joint point information;

[0149] Based on the salient features and mean characteristics , generating each limb part The corresponding weights are:

[0150]

[0151] In the formula, Indicates body parts The corresponding weight, represents concatenation, MLP represents the MLP encoding method / model, is the activation function, and n represents the total number of limb parts.

[0152] Preferably, in some embodiments, the sequence encoding specifically includes:

[0153] Constructing the action set at time t in the limb rehabilitation exercise video :

[0154]

[0155] In the formula, Indicates the limb part at time t The action signal, , Represents a complete action cycle;

[0156] Constructing a complete action cycle Action Set :

[0157]

[0158] In the formula, ; express The action set of the moment;

[0159] From the action set Filter the key actions related to action evaluation and build a key action set:

[0160]

[0161] In the formula, ;here, It is not necessarily a sequential moment, but corresponds to the moment of the key action of screening;

[0162] Encode the action sequence:

[0163]

[0164] In the formula, is the characteristic constant, determined in The coding width near the sequence is widened. is the normalization constant.

[0165] Further, in one of the embodiments, the evaluation model in step 4 includes a feature extraction module, a feature fusion module, a first mapping module and a second mapping module;

[0166] The feature extraction module includes: a first feature extraction unit for respectively extracting time features of equivalent shape information, vector information and joint point information; a second feature extraction unit for respectively extracting space features of equivalent shape information, vector information and joint point information;

[0167] The feature fusion module includes: a first feature fusion unit for fusing the time features of equivalent shape information, vector information and joint point information; a second feature fusion unit for fusing the spatial features of equivalent shape information, vector information and joint point information;

[0168] The first mapping module is used to map the fused spatial features and temporal features;

[0169] The second mapping module is used to establish a relationship between the features mapped by the first mapping module and the evaluation results.

[0170] Here, the relationship is established by means of a lookup table (a relationship table between features and evaluation results established in advance), so the evaluation results can be output directly.

[0171] Preferably, in some embodiments, combined Figure 3 The second feature extraction unit extracts the spatial features of equivalent shape information, vector information and joint point information respectively, specifically including:

[0172] (1) Spatial feature extraction of equivalent shape information

[0173] For every body part , input the equivalent shape information of the object to be evaluated into the CNN model, and extract the equivalent shape information features At the same time, the equivalent shape information of the standard rehabilitation action is input into the CNN model to extract the standard equivalent shape information features ;

[0174] Counting limb parts The corresponding equivalent shape information loss function :

[0175]

[0176] In the formula, is the activation function;

[0177] Based on the weights corresponding to each limb part obtained in step 3, the spatial features of the equivalent shape information of the whole body are obtained. :

[0178]

[0179] In the formula, is the activation function, Indicates body parts The corresponding weights, n represents the total number of limb parts;

[0180] (2) Spatial feature extraction of vector information

[0181] For every body part , input the vector information of the object to be evaluated into the CNN model and extract the vector information features At the same time, the vector information of the standard rehabilitation action is input into the CNN model to extract the standard vector information features ;

[0182] Counting limb parts The corresponding vector information loss function :

[0183]

[0184] Based on the weights corresponding to each limb part obtained in step 3, the spatial characteristics of the vector information of the whole body are obtained. :

[0185]

[0186] (3) Spatial feature extraction of joint point information

[0187] For every body part , input the joint point information of the object to be evaluated into the CNN model and extract the joint point features At the same time, the joint point information of the standard rehabilitation action is input into the CNN model to extract the standard joint point information features ;

[0188] Counting limb parts Corresponding joint point information loss function :

[0189]

[0190] Based on the weights corresponding to each limb part obtained in step 3, the spatial features of the joint point information of the whole body are obtained. :

[0191]

[0192] Preferably, in some embodiments, combined Figure 4 The first feature extraction unit extracts the time features of the equivalent shape information, the vector information and the joint point information respectively, specifically including:

[0193] (1) At time t, move each limb part The equivalent shape information / vector information / joint point information are input into the TCN module respectively:

[0194]

[0195] Among them, the TCN module includes a fully connected convolution for feature alignment , and the time interval between multiple layers of input is set by the kernel function step size and convolution parameters Modules; Indicates the limb part at time t Equivalent shape information / vector information / joint point information; Represents the temporal characteristics of the equivalent shape information / vector information / joint point information output by the TCN module;

[0196] Based on the weights corresponding to each limb part obtained in step 3, the time features of the equivalent shape information / vector information / joint point information of all limb parts at time t output after the TCN module are fused to obtain :

[0197]

[0198] make ;

[0199] (2) Based on the process of (1) above, we obtain the fusion feature set output by the TCN module within a complete action cycle T. :

[0200]

[0201] In the formula, ;

[0202] (3) Calculate the degree of attention paid to the key action time node encoding at time j at time t:

[0203]

[0204] in,

[0205]

[0206] In the formula, represents the weight, which is used to indicate the degree of attention paid to the key action time node encoding at time j at time t.

[0207] is the key action sequence encoding at time j calculated based on step 3, and d is Dimensions;

[0208] (4) Using the weights and Update fusion feature set get :

[0209]

[0210] in,

[0211]

[0212] Preferably, in some embodiments, combined Figure 5 The process of the first feature fusion unit / the second feature fusion unit performing feature fusion specifically includes:

[0213] After the temporal features / spatial features of the joint point information and vector information are respectively passed through the attention model ATT, a new first feature sequence is generated through addition operation;

[0214] The new first feature sequence and the temporal features / spatial features of the equivalent shape information are respectively subjected to the attention model ATT, and then a new second feature sequence is generated through an addition operation.

[0215] The operation of the attention model ATT is specifically as follows:

[0216]

[0217]

[0218] In the formula, , are the characteristic sequences of the corresponding channels, , , , is the learned projection matrix, is the dimension of the key-value set, Q and K are the learned query and key vectors, is the tensor dimension space corresponding to the input feature channel.

[0219] Preferably, in some embodiments, the first mapping module maps the fused spatial features and temporal features, specifically by inputting the fused spatial features and temporal features into a fully connected layer to establish a feature mapping.

[0220] In one embodiment, a multi-scale motor rehabilitation assessment system is provided, the system comprising:

[0221] The first module is used to collect limb rehabilitation exercise videos of the subject to be evaluated during rehabilitation training;

[0222] The second module is used to extract feature information of limb parts for each frame image in the limb rehabilitation exercise video, including equivalent shape information, vector information and joint point information;

[0223] The third module is used to perform feature encoding on each limb part and generate the weight and action sequence encoding corresponding to each limb part;

[0224] The fourth module is used to construct an evaluation model based on the feature encoding result, which is a temporal model that establishes the internal connection of actions and outputs the evaluation result of the health training of the object to be evaluated.

[0225] For the specific limitations of the multi-scale motion rehabilitation assessment system, please refer to the limitations of the multi-scale motion rehabilitation assessment method above, which will not be repeated here. Each module in the above-mentioned multi-scale motion rehabilitation assessment system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0226] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0227] Step 1, collecting limb rehabilitation exercise videos of the subject to be evaluated during rehabilitation training;

[0228] Step 2, for each frame of the limb rehabilitation exercise video, extracting feature information of the limb parts, including equivalent shape information, vector information and joint point information;

[0229] Step 3, feature encoding of each limb part, generating weights and action sequence encoding corresponding to each limb part;

[0230] Step 4: Based on the feature encoding result, an evaluation model is constructed, where the model is a temporal model that establishes the internal connection of actions, and outputs the evaluation result of the health training of the object to be evaluated.

[0231] For the specific limitations of each step, please refer to the limitations of the multi-scale motor rehabilitation assessment method above, which will not be repeated here.

[0232] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements:

[0233] Step 1, collecting limb rehabilitation exercise videos of the subject to be evaluated during rehabilitation training;

[0234] Step 2, for each frame of the limb rehabilitation exercise video, extracting feature information of the limb parts, including equivalent shape information, vector information and joint point information;

[0235] Step 3, feature encoding of each limb part, generating weights and action sequence encoding corresponding to each limb part;

[0236] Step 4: Based on the feature encoding result, an evaluation model is constructed, where the model is a temporal model that establishes the internal connection of actions, and outputs the evaluation result of the health training of the object to be evaluated.

[0237] For the specific limitations of each step, please refer to the limitations of the multi-scale motor rehabilitation assessment method above, which will not be repeated here.

[0238] It should be noted that the CNN model and the like used in the present invention are preferred, but are not limited to these models, and the application of other models also falls within the protection scope of the present invention.

[0239] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-scale motor rehabilitation assessment method, characterized in that: The method comprises the following steps: Step 1, collecting limb rehabilitation exercise videos of the subject to be evaluated during rehabilitation training; Step 2, for each frame of the limb rehabilitation exercise video, extracting feature information of the limb parts, including equivalent shape information, vector information and joint point information; Step 3, feature encoding of each limb part, generating weights and action sequence encoding corresponding to each limb part; Step 4: construct an evaluation model based on the feature encoding result, which is a temporal model that establishes the internal connection of actions and outputs the evaluation result of the rehabilitation training of the object to be evaluated.

2. The multi-scale motor rehabilitation assessment method according to claim 1, characterized in that: The limb part division in step 2, as well as the equivalent shape information, vector information and joint point information are specifically as follows: Right hip p1: equivalent shape rect1, vector v1, joint point (j0, j1); Right thigh p2: equivalent shape rect2, vector v2, joint point (j1, j2); Right calf p3: equivalent shape rect3, vector v3, joint point (j2, j3); Left hip p4: equivalent shape rect4, vector v4, joint point (j0, j4); Left thigh p5: equivalent shape rect5, vector v5, joint point (j4, j5); Left calf p6: equivalent shape rect6, vector v6, joint point (j5, j6); Torso p7: equivalent shape rect7, vector v7, joint points (j0, j7, j8); Neck p8: equivalent shape rect8, vector v8, joint points (j8, j9, j 10 ); Left shoulder p9: equivalent shape rect9, vector v9, joint point (j8, j 11 ); Left rear arm 10 : Equivalent shape rect 10 , vector v 10 、Joint point(j 11 , j 12 ); Left forearm 11 : Equivalent shape rect 11 , vector v 11 、Joint point(j 12 , j 13 ); Right shoulder 12 : Equivalent shape rect 12 , vector v 12 , joint point (j8, j 14 ); Right forearm 13 : Equivalent shape rect 13 , vector v 13 、Joint point(j 14 , j 15 ); Right rear arm 14 : Equivalent shape rect 14 , vector v 14 、Joint point(j 15 , j 16 ).

3. The multi-scale motor rehabilitation assessment method according to claim 2, characterized in that: The feature encoding in step 3 includes: Position encoding, used to generate weights corresponding to limb parts; Sequence coding is used for rehabilitation training action sequence coding.

4. The multi-scale motor rehabilitation assessment method according to claim 3, characterized in that: The position code specifically includes: Each limb part is obtained by maximum pooling and mean pooling operations respectively. Features The salient features and mean characteristics : ； , ； In the formula, , Represents the maximum pooling and mean pooling operations respectively, and N×M is the feature The spatial dimension of Features including equivalent shape information, vector information, and joint point information; Based on the salient features and mean characteristics , generating each limb part The corresponding weights are: ； In the formula, Indicates body parts The corresponding weight, represents concatenation, MLP represents the MLP encoding method / model, is the activation function, and n represents the total number of limb parts.

5. The multi-scale motor rehabilitation assessment method according to claim 3, characterized in that: The sequence encoding specifically includes: Constructing the action set at time t in the limb rehabilitation exercise video : ； In the formula, Indicates the limb part at time t action signal, , Represents a complete action cycle; Constructing a complete action cycle Action Set : ； In the formula, ; express The action set of the moment; From the action set Filter the key actions related to action evaluation and build a key action set: ； In the formula, ;here, It is not necessarily a sequential moment, but corresponds to the moment of the key action of screening; Encode the action sequence: ； In the formula, is the characteristic constant, determined in The coding width near the sequence is widened. is the normalization constant.

6. The multi-scale motor rehabilitation assessment method according to claim 1, characterized in that: The evaluation model in step 4 includes a feature extraction module, a feature fusion module, a first mapping module and a second mapping module; The feature extraction module includes: a first feature extraction unit for respectively extracting time features of equivalent shape information, vector information and joint point information; a second feature extraction unit for respectively extracting space features of equivalent shape information, vector information and joint point information; The feature fusion module includes: a first feature fusion unit for fusing the time features of equivalent shape information, vector information and joint point information; a second feature fusion unit for fusing the spatial features of equivalent shape information, vector information and joint point information; The first mapping module is used to map the fused spatial features and temporal features; The second mapping module is used to establish a relationship between the features mapped by the first mapping module and the evaluation results.

7. The multi-scale motor rehabilitation assessment method according to claim 6, characterized in that: The second feature extraction unit extracts the spatial features of equivalent shape information, vector information and joint point information respectively, specifically including: (1) Spatial feature extraction of equivalent shape information For every body part , input the equivalent shape information of the object to be evaluated into the CNN model, and extract the equivalent shape information features At the same time, the equivalent shape information of the standard rehabilitation action is input into the CNN model to extract the standard equivalent shape information features ; Counting limb parts The corresponding equivalent shape information loss function : ； In the formula, is the activation function; Based on the weights corresponding to each limb part obtained in step 3, the spatial features of the equivalent shape information of the whole body are obtained. : ； In the formula, is the activation function, Indicates body parts The corresponding weights, n represents the total number of limb parts; (2) Spatial feature extraction of vector information For every body part , input the vector information of the object to be evaluated into the CNN model and extract the vector information features At the same time, the vector information of the standard rehabilitation action is input into the CNN model to extract the standard vector information features ; Counting limb parts The corresponding vector information loss function : ； Based on the weights corresponding to each limb part obtained in step 3, the spatial characteristics of the vector information of the whole body are obtained. : ； (3) Spatial feature extraction of joint point information For every body part , input the joint point information of the object to be evaluated into the CNN model and extract the joint point features At the same time, the joint point information of the standard rehabilitation action is input into the CNN model to extract the standard joint point information features ; Counting limb parts Corresponding joint point information loss function : ； Based on the weights corresponding to each limb part obtained in step 3, the spatial features of the joint point information of the whole body are obtained. : 。 8. The multi-scale motor rehabilitation assessment method according to claim 6, characterized in that: The first feature extraction unit extracts time features of equivalent shape information, vector information and joint point information respectively, specifically including: (1) At time t, move each limb part The equivalent shape information / vector information / joint point information are input into the TCN module respectively: ； Among them, the TCN module includes a fully connected convolution for feature alignment , and the time interval between multiple layers of input is set by the kernel function step size and convolution parameters Modules; Indicates the limb part at time t Equivalent shape information / vector information / joint point information; Represents the temporal characteristics of the equivalent shape information / vector information / joint point information output by the TCN module; Based on the weights corresponding to each limb part obtained in step 3, the time features of the equivalent shape information / vector information / joint point information of all limb parts at time t output after the TCN module are fused to obtain : ； make ; (2) Based on the process of (1) above, we obtain the fusion feature set output by the TCN module within a complete action cycle T. : ； In the formula, ; (3) Calculate the degree of attention paid to the key action time node encoding at time j at time t: ； in, ； In the formula, represents the weight, which is used to indicate the degree of attention paid to the key action time node encoding at time j at time t. is the key action sequence encoding at time j calculated based on step 3, and d is Dimensions; (4) Using the weights and Update fusion feature set get : ； in, 。 9. The multi-scale motor rehabilitation assessment method according to claim 6, characterized in that: The process of the first feature fusion unit / the second feature fusion unit performing feature fusion specifically includes: After the temporal features / spatial features of the joint point information and vector information are respectively passed through the attention model ATT, a new first feature sequence is generated through addition operation; The new first feature sequence and the temporal features / spatial features of the equivalent shape information are respectively subjected to the attention model ATT, and then a new second feature sequence is generated through an addition operation.

10. The multi-scale motor rehabilitation assessment method according to claim 6, characterized in that: The first mapping module maps the fused spatial features and temporal features, specifically by inputting the fused spatial features and temporal features into a fully connected layer to establish a feature mapping.

Images