A system and method for assessing limb dysfunction in a stroke patient

Abstract

The application is a kind of system and method for evaluating the limb dysfunction of stroke patients, belonging to the field of rehabilitation medicine. The system is composed of a data acquisition module, a data storage module, a data preprocessing module, a feature extraction module, a feature fusion module, a prediction module and an output module. The method includes: S1: acquisition of video data and text data; S2: data preprocessing to obtain a spatiotemporal skeleton graph and scale features; S3: constructing a scale feature set using a KNN network; S4: extracting image features and limb node features using a C3D network and a GCN network; S5: feature fusion to obtain limb node fusion features; S6: training the prediction module; S7: using a recommendation system to give a scale feature expression; S8: using a classifier to evaluate the limb dysfunction of the patient. The method can avoid the interference of subjective cognition, accurately evaluate the limb dysfunction of stroke patients, and assist doctors in developing subsequent rehabilitation treatment plans for patients.

Description

TECHNICAL FIELD

[0001] The present application relates to a limb dysfunction assessment system and method for stroke patients, belonging to the field of rehabilitation medicine, and in particular to the assessment of limb dysfunction in stroke patients. BACKGROUND

[0002] Stroke is a common cerebrovascular disease with high morbidity, mortality and disability rate, and has become the leading cause of disability in Chinese adults. About 70% to 80% of stroke patients will develop hemiplegia, mainly manifested as unilateral limb dysfunction. Assessing the limb dysfunction of stroke patients is crucial for developing personalized rehabilitation programs, evaluating treatment effectiveness and improving the quality of life of patients.

[0003] Currently, the assessment of limb dysfunction in stroke patients in clinical practice mainly relies on scales, physical examinations and imaging examinations. Compared with complex, indirect and expensive physical examinations and imaging examinations, scale methods are more direct and efficient. Commonly used assessment scales include the Activity of Daily Living Scale (ADL), the Modified Rivermead Mobility Index Scale (MRMI) and the Berg Balance Scale.

[0004] These scales reflect the functional status of hemiplegic patients from different angles, but ultimately a comprehensive judgment and assessment is made by clinicians based on their clinical experience. The assessment results are heavily dependent on the experience of doctors, and there are strong subjectivity, non-uniform assessment methods and lack of evaluation standards. This inevitably seriously affects the correctness and reliability of the evaluation of the degree of limb dysfunction in hemiplegic patients, especially in remote areas and rural areas where medical resources are scarce. Therefore, the automatic assessment of limb dysfunction in hemiplegic patients is an urgent problem to be solved.

[0005] With the development of artificial intelligence technology, machine learning (ML) is gradually applied to the assessment of limb dysfunction in stroke patients. Machine learning algorithms can learn information from data, make predictions and decisions, and have the advantages of objectivity and automation. The invention patent "A method for automatic assessment of limb dysfunction of hemiplegic patients" (CN114533043A) can realize accurate assessment of the upper and lower limb function levels of patients by constructing a SVM classification evaluation model based on data-driven. It provides personalized rehabilitation programs for stroke patients and improves their quality of life. However, the scale data in this method is still obtained by combining the subjective experience of doctors with the text description of the scale, which has individual cognitive bias. At the same time, the data features expressed in text form are difficult to express the spatial and temporal characteristics of patient limb movement, and there is a bias in feature expression accuracy. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the prior art and provide a limb dysfunction assessment system and method for stroke patients. First, the K-Nearest Neighbor (KNN) algorithm is used to find the adjacent features of the patient's scale features and construct a candidate space for the scale features. Then, combined with video data, the deep learning network is used to obtain image features and limb node features to get limb node fusion features. The recommendation system is used to filter out the scale features closest to the limb node fusion features in the candidate space as the final scale feature expression. Then, the classifier is used to classify the scale feature expression. Finally, the interference of subjective cognition on the features is avoided, and the dimensionality information of the features is enriched.

[0007] To achieve the above purpose, the present application provides the following technical solutions:

[0008] The application discloses a system for evaluating limb dysfunction of a stroke patient, which is characterized by comprising a data acquisition module, a data storage module, a data preprocessing module, a feature extraction module, a feature fusion module, a prediction module and an output module; the data acquisition module is used for acquiring video data shot by a camera and text data of a patient scale; the data storage module comprises a video data buffer unit, a text data buffer unit and a feature storage unit, and is used for storing data and features; the data preprocessing module comprises a joint extraction module and an encoding and normalization module, and is used for preprocessing the video data and the text data; the feature extraction module comprises a KNN network, a graph convolutional network (GCN) and a C3D network, and is used for extracting features and constructing a feature space; the prediction module comprises a recommendation system and a classifier, and is used for feature screening and classification; the output module is an output interface, and is used for feeding back a limb dysfunction evaluation result to a user; the text data buffer unit is connected with the data acquisition module and the encoding and normalization module respectively; the video data buffer unit is connected with the data acquisition module, the joint extraction module and an input of the C3D network respectively; the feature storage unit is connected with the feature extraction module; an input and an output of the KNN network are connected with an output of the encoding and normalization module and an input of the recommendation system respectively; an input and an output of the GCN network are connected with an output of the joint extraction module and an input of the feature fusion module respectively; an output of the C3D network is connected with an input of the feature fusion module; an output of the feature fusion module is connected with an input of the recommendation system; an output of the recommendation system is connected with an input of the classifier; and an output of the classifier is connected with the output module.

[0009] Further, the recommendation system is a reinforcement learning network, a state of the reinforcement learning network is an output of the feature fusion module, an action space of the reinforcement learning network is an output of the KNN network, and a reward is a percentage of a limb dysfunction evaluation result output by the classifier being correct.

[0010] Further, the camera is a Kinect device; and the joint extraction module is a Kinect software development tool, which is used for acquiring positions and connection relationships of joint nodes of a human body in an image.

[0011] Further, the feature fusion module is a cross-attention mechanism, which is used for focusing on image features and limb node features.

[0012] Further, the classifier is composed of a full connection network and a Softmax layer.

[0013] Further, the feature extraction module, the feature fusion module and the prediction module are all 2 in number, and the upper limbs and lower limbs of the patient are respectively evaluated for limb dysfunction.

[0014] A limb dysfunction evaluation method for stroke patients, characterized in that it comprises the following steps:

[0015] S1: input the video data of the patient's movement collected by the camera and the text data of the patient's scale into the data acquisition module, and store them in the corresponding positions of the storage module;

[0016] S2: the data preprocessing module respectively preprocesses the video data and the text data to obtain the space-time skeleton graph and the scale feature;

[0017] S3: use the KNN network to filter out a number of scale features closest to the patient's scale feature in the historical data to construct a scale feature set;

[0018] S4: use the C3D network to extract the image features of the video data, and use the GCN network to extract the limb node features of the space-time skeleton graph;

[0019] S5: use the feature fusion module to fuse the image features and the limb node features to obtain the limb node fusion features;

[0020] S6: use the historical data to train the network in the prediction module with the Brunnstrom scale impairment level division as the label;

[0021] S7: input the limb node fusion features into the trained recommendation system, and use the recommendation system to give a scale feature closest to the limb node fusion features as the scale feature expression;

[0022] S8: input the scale feature expression into the trained classifier to realize the evaluation of the patient's limb dysfunction severity.

[0023] Further, the text data of the scale includes ADL scale, MRMI scale and Berg scale, and is stored in a list format; wherein the upper limb related part of the ADL scale reflects the patient's eating, bed-chair transfer, toilet, dressing, clothing, and bathing; the lower limb related part of the ADL scale reflects the patient's walking, up and down stairs, bed-chair transfer, toilet, controlling defecation, bathing, and controlling urination; the upper limb related part of the MRMI scale reflects the patient's turning over in bed, bed-chair transfer, dressing, clothing, and bathing; the lower limb related part of the MRMI scale reflects the patient's walking, bed-chair transfer, up and down stairs, bathing, and controlling urination; and the lower limb related part of the Berg scale reflects the patient's sitting to standing, independent standing, standing to sitting, bed-chair transfer, stepping on the stairs with two feet alternately, standing with two feet forward and backward, and standing on one leg.

[0024] Further, the step S2 specifically includes the following steps.

[0025] S201: extracting a spatio-temporal skeleton graph from the video data by using a joint extraction module;

[0026] S202: extracting scale features from the text data by using an encoding and normalization module.

[0027] Further, the step S201 includes the following steps.

[0028] S2011: acquiring human skeleton information in the video image frame by frame;

[0029] The human skeleton information includes two-dimensional coordinate point information or three-dimensional coordinate point information and confidence of the human skeleton key node , wherein, is the skeleton node spatial coordinate, is the confidence of the skeleton detection; and

[0030] S2012: constructing a spatio-temporal skeleton graph composed of a node (N) set and a connection relationship (L) set ; wherein, the frame number is K frames, and the number of skeleton key nodes is S; the connection relationship set has two types, one type of connection relationship set is the natural connection of the skeleton nodes in the same frame, which can be expressed as , and the other type of connection relationship set is the connection of the same node on consecutive frames, which can be expressed as .

[0031] Further, the step S202 includes the following steps.

[0032] S2021: encoding the text data list by using one-hot encoding;

[0033] S2022: normalize the encoding to obtain a scale feature.

[0034] Further, according to the movement condition, the Brunnstrom scale is divided into I, II, III, IV, V and VI stages for upper limb and lower limb dysfunction of stroke patients, and the six stages mark the decrease of limb motor dysfunction in turn; the output of the corresponding classifier is 6-dimensional, and the output value corresponds to the probability value of the Brunnstrom scale level of limb dysfunction. Generally, the level with the maximum probability value is taken as the output of the limb dysfunction evaluation result.

[0035] Further, the loss function for training the network of the prediction module in step S6 is a cross-entropy loss function.

[0036] The beneficial effects of the present application are: a limb dysfunction evaluation system and method for stroke patients is provided, which is based on the scale text data of subjective judgment of doctors, and a KNN network is used to construct a plurality of scale feature sets of patients, and the limb node features and image features of image data are used to eliminate the error of subjective judgment, the accuracy of the corresponding features of the image and the accuracy of the finally selected scale features are ensured through the cross-attention mechanism and the recommendation system, and the accurate Brunnstrom scale rating prediction of the limb dysfunction of stroke patients is realized, which assists the doctors in formulating the subsequent rehabilitation treatment plan for the patients, avoids the interference of subjective cognition of doctors on the judgment result, and enriches the dimension information of the features. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to make the purpose, technical scheme of the present application more clear and clear, the present application provides the following drawings for description:

[0038] Figure 1 It is the architecture diagram of a limb dysfunction evaluation system for stroke patients in embodiment 1 of the present application;

[0039] Figure 2 It is a flowchart of a limb dysfunction evaluation method for stroke patients in embodiment 2 of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose and technical scheme of the present application more clear and clear, the present application will be described in detail below in combination with the drawings and embodiments.

[0041] Embodiment 1: the effect of recovery of stroke patients needs to be accurately and conveniently evaluated to guide the rehabilitation training plan in the later period. In order to avoid the judgment deviation caused by subjective cognition of different doctors and avoid the complex physical examination process, the present embodiment provides a limb dysfunction evaluation system for stroke patients.

[0042] Combining Figure 1 , data acquisition module (1), data storage module (2), data preprocessing module (3), feature extraction module (4), feature fusion module (5), prediction module (6) and output module (7) constitute;The data acquisition module (1) is used for collecting video data and text data of patient scale shot by camera;The data storage module (2) includes video data buffer unit (21), text data buffer unit (22) and feature storage unit (23), which is used for storing data and features;The data preprocessing module (3) includes joint extraction module (31) and encoding and normalization module (32), which is used for preprocessing video data and text data;The feature extraction module (4) includes KNN network (41), GCN network (42) and C3D network (43), which is used for extracting features and constructing feature space;The prediction module (6) includes recommendation system (61) and classifier (62), which is used for feature screening and classification;The output module (7) is an output interface, which is used for feeding back the results of limb function assessment to the user;The text data buffer unit (22) is connected with data acquisition module (1) and encoding and normalization module (32) respectively;The video data buffer unit (21) is connected with data acquisition module (1), joint extraction module (31) and input of C3D network (43) respectively;The feature storage unit (23) is connected with feature extraction module (4);The input and output of KNN network (41) are connected with the output of encoding and normalization module (32) and the input of recommendation system (61) respectively;The input and output of GCN network (42) are connected with the output of joint extraction module (31) and the input of feature fusion module (5) respectively;The output of C3D network (43) is connected with the input of feature fusion module (5);The output of feature fusion module (5) is connected with the input of recommendation system (61);The output of recommendation system (61) is connected with the input of classifier (62);The output of classifier (62) is connected with output module (7).

[0043] Further, the recommendation system (61) is a reinforcement learning network, the state of which is the output of feature fusion module (5), the action space of which is the output of KNN network (41), and the reward is the percentage of correct limb function assessment results output by the classifier (62).

[0044] Further, the camera is Kinect device;The joint extraction module is Kinect software development tool, which is used for obtaining the position and connection relationship of human joint nodes in image.

[0045] Further, the feature fusion module (5) is cross attention mechanism, which is used for focusing on image features and limb node features.

[0046] Further, the classifier (62) is a full connection network in series with a Softmax layer.

[0047] Further, the feature extraction module (4), the feature fusion module (5), and the prediction module (6) are all 2 in number, and are used to respectively evaluate the upper and lower limbs of the patient for limb dysfunction.

[0048] In operation, the collected image data and text data are input into the data acquisition module (1) and the data storage module (2) of the system, and are then respectively preprocessed by the preprocessing module (3) to extract the spatiotemporal skeleton graph and encode, normalize, etc., and then the feature extraction module (4) is used to extract features, the feature fusion module (5) is used to focus on the node features and image features of the image data, and finally the prediction module (6) is used to evaluate the limb dysfunction of the stroke patient, and the output module (7) is used to feed back to the doctor to assist the doctor in specifying the rehabilitation plan.

[0049] Embodiment 2: In combination with Embodiment 1, the postoperative effect of the stroke patient needs to be accurately and conveniently evaluated to guide the rehabilitation training plan in the later stage. In order to avoid the judgment deviation caused by the subjective cognition of different doctors and avoid the complex physical examination process, the present embodiment provides "a limb dysfunction evaluation method for stroke patients".

[0050] In combination with Figure 2 , the following steps are included:

[0051] Step 1: screening out the patient scale commonly used in clinical practice, i.e. ADL scale, mobility assessment MRMI scale, Berg scale, and Brunnstrom scale, storing them in a list format as text data; and using a Kinect device to collect the motion video of the stroke patient and storing it frame by frame as video data.

[0052] Step 2: the data acquisition module (1) collects the text data and video data and stores them in the corresponding positions of the storage module (2).

[0053] Step 3: the data preprocessing module (3) respectively pre-processes the video data and the text data to obtain the spatiotemporal skeleton graph and the scale features. Specifically:

[0054] 1) using the joint node extraction module (31) to extract the spatiotemporal skeleton graph from the video data;

[0055] 2) using the encoding and normalization module (32) to extract the scale features from the text data.

[0056] Further, step 1) includes the following steps:

[0057] 1-1) Obtain human skeleton information in a video image frame by frame;

[0058] The human skeleton information includes two-dimensional coordinate point information or three-dimensional coordinate point information and self-confidence of a human skeleton key node , wherein, is a skeleton node spatial coordinate, is a self-confidence when the skeleton is detected;

[0059] 1-2) Construct a space-time skeleton graph composed of a node (N) set and a connection relationship (L) set ; wherein, , the frame number is K frames, and the number of skeleton key nodes is S; the connection relationship set has two types, one connection relationship set is the natural connection of the skeleton nodes in the same frame, which can be represented as , and the other connection relationship set is the connection of the same node on consecutive frames, which can be represented as .

[0060] Further, the step 2) includes the following steps:

[0061] 2-1) Encode the text data list using one-hot encoding;

[0062] 2-2) Normalize the encoding to obtain scale characteristics. The normalization is scaled to the [0, 1] interval using MaxMinScaler.

[0063] Step four, use KNN network (41) to filter out K scale characteristics closest to the patient scale characteristics in the historical data, and construct a scale characteristic set.

[0064] Step five, use C3D network (43) to extract image features of video data, and use GCN network (42) to extract limb node features of space-time skeleton graph.

[0065] Step six, use the feature fusion module (5) to fuse the image features and the limb node features to obtain the limb node fusion features.

[0066] Step seven, use the Brunnstrom scale obstacle level division as a label, and train the network in the prediction module (6) using historical data.

[0067] The output of the classifier (62) in the prediction module (6) is 6-dimensional, and the output value corresponds to the probability value of the level of the Brunnstrom scale of the limb dysfunction.

[0068] The loss function of the network training of the prediction module (6) is a cross-entropy loss function.

[0069] During the training, the network in the prediction module can also be trained jointly with the KNN network (41), the GCN network (42) and the C3D network (43).

[0070] Step eight, input the limb node fusion feature into the trained recommendation system (61), and use the recommendation system (61) to give a scale feature closest to the limb node fusion feature as the scale feature expression.

[0071] Step nine, input the scale feature expression into the trained classifier (62) to realize the assessment of the severity of the patient's limb dysfunction, take the grade of the Brunnstrom scale with the maximum probability value as the assessment result of the limb dysfunction, and use the output module (7) to feedback to the doctor to assist the doctor in specifying the rehabilitation plan.

[0072] Finally, it should be pointed out that the above preferred embodiments are only used to illustrate the technical solutions of the present application and not to limit it, although the present application has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present application.

Claims

1. A system for evaluating a stroke patient's limb dysfunction, characterized by, The application relates to a limb function disorder assessment system, which is composed of a data acquisition module (1), a data storage module (2), a data preprocessing module (3), a feature extraction module (4), a feature fusion module (5), a prediction module (6) and an output module (7); the data acquisition module (1) is used for collecting video data shot by a camera and text data of a patient scale; the data storage module (2) comprises a video data buffer unit (21), a text data buffer unit (22) and a feature storage unit (23) and is used for storing data and features; the data preprocessing module (3) comprises a joint point extraction module (31) and an encoding and normalization module (32) and is used for preprocessing the video data and the text data; the feature extraction module (4) comprises a K-Nearest Neighbor (KNN) network (41), a Graph Convolutional Network (GCN) network (42) and a C3D (3D convolution) network (43) and is used for extracting features and constructing a feature space; the prediction module (6) comprises a recommendation system (61) and a classifier (62) and is used for feature screening and classification; the output module (7) is an output interface and is used for feeding back a limb function disorder assessment result to a user; the text data buffer unit (22) is connected with the data acquisition module (1) and the encoding and normalization module (32) respectively; the input of the video data buffer unit (21) is connected with the output of the data acquisition module (1), the output of the video data buffer unit (21) is connected with the input of the joint point extraction module (31) and the C3D network (42); the feature storage unit (23) is connected with the feature extraction module (4); the input and the output of the KNN network (41) are connected with the output of the encoding and normalization module (32) and the input of the recommendation system (61) respectively; the input and the output of the GCN network (42) are connected with the output of the joint point extraction module (31) and the input of the feature fusion module (5) respectively; the output of the C3D network (43) is connected with the input of the feature fusion module (5); the output of the feature fusion module (5) is connected with the input of the recommendation system (61); the output of the recommendation system (61) is connected with the input of the classifier (62); the output of the classifier (62) is connected with the output module (7); the feature fusion module (5) is a cross attention mechanism and is used for focusing image features and limb node features; the recommendation system (61) is a reinforcement learning network, the state of the recommendation system (61) is the output of the feature fusion module (5), the action space of the recommendation system (61) is the output of the KNN network (41), and the reward is the percentage of correct limb function disorder assessment results output by the classifier (62).

2. The system for evaluating the disability of a stroke patient's limb according to claim 1, wherein the camera is a Kinect device; the joint point extraction module is a Kinect software development tool and is used for acquiring the position and the connection relationship of human joint nodes in an image.

3. The system for assessing the disability of a stroke patient's limb according to claim 1, wherein The classifier (62) is a full connection network in series with a Softmax layer.

4. The system for assessing the disability of a stroke patient's limb according to claim 1, wherein The number of the feature extraction module (4), the feature fusion module (5) and the prediction module (6) is 2, and the upper limbs and lower limbs of the patient are respectively evaluated for the limb dysfunction.

5. The method for evaluating the functional disability of a stroke patient according to any one of claims 1 to 4, characterized in that, The method comprises the following steps: S1: input the video data of the patient's movement collected by the camera and the text data of the patient's scale into the data acquisition module (1), and store them in the corresponding positions of the storage module (2); S2: the data preprocessing module (3) respectively preprocesses the video data and the text data to obtain a space-time skeleton graph and scale features; S3: use the KNN network (41) to filter out a plurality of scale features closest to the patient's scale features from the historical data to construct a scale feature set; S4: use the C3D network (43) to extract image features of the video data, and use the GCN network (42) to extract limb node features of the space-time skeleton graph; S5: use the feature fusion module (5) to fuse the image features and the limb node features to obtain limb node fusion features; S6: use the historical data to train the network in the prediction module (6) by taking the Brunnstrom scale obstacle level division as a label; S7: input the limb node fusion features into the trained recommendation system (61), and use the recommendation system (61) to give a scale feature closest to the limb node fusion features as a scale feature expression; S8: input the scale feature expression into the trained classifier (62) to evaluate the severity of the patient's limb dysfunction.

6. The method of claim 5, wherein the method is for assessing the degree of disability of a stroke patient. The text data of the scale comprises ADL scale, MRMI scale and Berg scale, and is stored in a list format; wherein the upper limb related part of the ADL scale reflects the patient's eating, bed-chair transfer, toilet, dressing, clothing, and bathing; the lower limb related part of the ADL scale reflects the patient's walking, stair climbing, bed-chair transfer, toilet, controlling defecation, bathing, and controlling urination; the upper limb related part of the MRMI scale reflects the patient's turning over in bed, bed-chair transfer, dressing, clothing, and bathing; the lower limb related part of the MRMI scale reflects the patient's walking, bed-chair transfer, stair climbing, bathing, and controlling urination; and the lower limb related part of the Berg scale reflects the patient's sitting to standing, independent standing, standing to sitting, bed-chair transfer, stepping on the stairs with both feet, standing with both feet, and standing on one leg.

7. The method of claim 5, wherein the method is used for assessing the functional disability of a stroke patient. The step S2 specifically comprises: S201: use the joint node extraction module (31) to extract a space-time skeleton graph from the video data; S202: use the encoding and normalization module (32) to extract scale features from the text data; The step S201 comprises the following steps: S2011: acquire human skeleton information in the video image frame by frame; the human skeleton information comprises coordinate point information and confidence of human skeleton key nodes; S2012: construct a space-time skeleton graph composed of a node set and a connection relationship set; The step S202 comprises the following steps: S2021: encode the text data list by using one-hot encoding; S2022: normalize the encoding to obtain the scale feature.

8. The method of claim 5, wherein the method is used for assessing the functional disability of a stroke patient. The Brunnstrom scale in step S6 respectively classifies the upper limb and lower limb dysfunction of the stroke patient into I phase, II phase, III phase, IV phase, V phase and VI phase, which respectively correspond to the 6 dimensions of the classifier (62) output, and the output value corresponds to the probability value of the Brunnstrom scale level of the limb dysfunction; the loss function of the network training of the prediction module (6) is a cross-entropy loss function.

Images