Limb dysfunction assessment system and method for stroke patient

Abstract

The invention relates to a limb dysfunction assessment system and method for a stroke patient, and belongs to the field of rehabilitation medicine. The system is composed of a data acquisition module, a data storage module, a data preprocessing module, a feature extraction module, a feature fusion module, a prediction module and an output module. The method comprises the following steps: S1, collecting video data and text data; s2, preprocessing the data to obtain a space-time skeleton diagram and scale features; s3, constructing a scale feature set by using a KNN network; s4, extracting image features and limb node features by using a C3D network and a GCN network; s5, carrying out feature fusion to obtain limb node fusion features; s6, training a prediction module; s7, using a recommendation system to give scale feature expression; and S8, evaluating the limb dysfunction of the patient by using the classifier. By means of the method, interference of subjective cognition can be avoided, accurate assessment of limb dysfunction of the stroke patient is achieved, and doctors are assisted in making subsequent rehabilitation treatment plans for the patient.

Description

Technical Field

[0001] The present invention relates to a system and method for evaluating limb dysfunction in stroke patients, belonging to the field of rehabilitation medicine, and in particular to evaluating limb dysfunction in stroke patients. Background Art

[0002] Stroke is a common cerebrovascular disease with high morbidity, mortality and disability rates, and has become the leading cause of disability among Chinese adults. About 70% to 80% of stroke patients will experience hemiplegia, mainly manifested as unilateral limb dysfunction. Assessing limb dysfunction in stroke patients is crucial for developing personalized rehabilitation programs, evaluating treatment effects and improving patients' quality of life.

[0003] At present, the clinical assessment of limb dysfunction in stroke patients mainly relies on scales, physical examinations, and imaging examinations. Compared with complex, indirect, and expensive methods such as physical examinations and imaging examinations, scale methods are more direct and efficient. Commonly used assessment scales include the Activity of Daily Living Scale (ADL), the Modified Rivermead Mobility Index Scale (MRMI), and the Berg Balance Scale.

[0004] These scales reflect the functional status of hemiplegic patients from different perspectives, but ultimately clinicians make comprehensive judgments and assessments based on their own clinical experience. The evaluation results are heavily dependent on the experience of doctors, and there are problems such as strong subjectivity, inconsistent evaluation methods, and lack of evaluation standards. This is bound to seriously affect the correctness and reliability of the evaluation of the degree of limb dysfunction in hemiplegic patients, especially in remote areas and rural areas with scarce medical resources. Therefore, the automatic evaluation of limb dysfunction in hemiplegic patients is an urgent problem to be solved.

[0005] With the development of artificial intelligence technology, machine learning (ML) has gradually been applied to the assessment of limb dysfunction in stroke patients. Machine learning algorithms can learn information from data, make predictions and decisions, and have the advantages of objectivity and automation. The invention patent "A method for automatic assessment of limb dysfunction in hemiplegic patients" (CN114533043A) can accurately assess the functional levels of the upper and lower limbs of patients by constructing a data-driven SVM classification and evaluation model, providing stroke patients with personalized rehabilitation plans and improving their quality of life. However, the scale data in this method is still obtained by combining the doctor's subjective experience with the text description of the scale, and there are individual cognitive biases. At the same time, data features expressed in text form alone are difficult to express the spatial and temporal characteristics of the patient's limb movements, and there are deviations in the accuracy of feature expression. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and provide a system and method for evaluating limb dysfunction in stroke patients. First, the K-nearest Neighbor (KNN) algorithm is used to find adjacent features of the patient's scale features, and a candidate space of the scale features is constructed based on this. Then, in combination with video data, a deep learning network is used to obtain image features and limb node features to obtain limb node fusion features. A recommendation system is used to screen out the scale features closest to the limb node fusion features in the candidate space as the final scale feature expression, and then a classifier is used to classify the scale feature expressions. Finally, the interference of subjective cognition on the features is avoided, and the dimensional information of the features is enriched at the same time.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A limb dysfunction assessment system for stroke patients, characterized in that it is composed of a data acquisition module, a data storage module, a data preprocessing module, a feature extraction module, a feature fusion module, a prediction module and an output module; the data acquisition module is used to collect video data captured by a camera and text data of a patient scale; the data storage module comprises a video data cache unit, a text data cache unit and a feature storage unit for storing data and features; the data preprocessing module comprises a joint point extraction module and an encoding and normalization module for preprocessing video data and text data; the feature fusion module comprises a KNN network, a Graph Convolutional Network (GCN) network, a C3D (3D convolution (C3D for short) network, used for extracting features and constructing feature space; the prediction module includes a recommendation system and a classifier, used for feature screening and classification; the output module is an output interface, used for feeding back the limb dysfunction assessment results to the user; the text data cache unit is respectively connected to the data acquisition module and the encoding and normalization module; the video data cache unit is respectively connected to the data acquisition module, the joint point extraction module, and the input of the C3D network; the feature storage unit is connected to the feature extraction module; the input and output of the KNN network are respectively connected to the output of the encoding and normalization module and the input of the recommendation system; the input and output of the GCN network are respectively connected to the output of the joint point extraction module and the input of the feature fusion module; the output of the C3D network is connected to the input of the feature fusion module; the output of the feature fusion module is connected to the input of the recommendation system; the output of the recommendation system is connected to the input of the classifier; the output of the classifier is connected to the output module.

[0009] Furthermore, the recommendation system is a reinforcement learning network, whose state is the output of the feature fusion module, whose action space is the output of the KNN network, and the reward is the correct percentage of the limb dysfunction assessment results output by the classifier.

[0010] Furthermore, the camera is a Kinect device; the joint extraction module is a Kinect software development tool, which is used to obtain the positions and connection relationships of human joint nodes in the image.

[0011] Furthermore, the feature fusion module is a cross-attention mechanism, which is used to focus on image features and limb node features.

[0012] Furthermore, the classifier is composed of a fully connected network connected in series with a Softmax layer.

[0013] Furthermore, the number of the feature extraction modules, feature fusion modules, and prediction modules is 2, which respectively evaluate the limb dysfunction of the patient's upper limbs and lower limbs.

[0014] A method for evaluating limb dysfunction in stroke patients, comprising the following steps:

[0015] S1: inputting the video data of the patient's movement collected by the camera and the text data of the patient's scale into the data acquisition module, and storing them in the corresponding position of the storage module;

[0016] S2: The data preprocessing module preprocesses the video data and text data to obtain the spatiotemporal skeleton graph and scale features respectively;

[0017] S3: Use the KNN network to filter out several scale features that are closest to the patient's scale features in the historical data and construct a scale feature set;

[0018] S4: Use C3D network to extract image features of video data, and use GCN network to extract limb node features of spatiotemporal skeleton graph;

[0019] S5: Using the feature fusion module to fuse the image features and the limb node features to obtain the limb node fusion features;

[0020] S6: Using the Brunnstrom scale obstacle level classification as labels, the network in the prediction module is trained using historical data;

[0021] S7: Input the limb node fusion feature into the trained recommendation system, and use the recommendation system to give a scale feature that is closest to the limb node fusion feature as the scale feature expression;

[0022] S8: Input the scale feature expression into the trained classifier to evaluate the severity of the patient's limb dysfunction.

[0023] Furthermore, the text data of the scale includes an ADL scale, an MRMI scale, and a Berg scale, which are stored in a list format; wherein the upper limb-related part of the ADL scale reflects the patient's eating, bed-chair transfer, going to the toilet, grooming, dressing, and bathing items; the lower limb-related part of the ADL scale reflects the patient's walking, going up and down stairs, bed-chair transfer, going to the toilet, controlling bowel movements, bathing, and controlling urination items; the upper limb-related part of the MRMI scale reflects the patient's turning over in bed, bed-chair transfer, grooming, dressing, and bathing items; the lower limb-related part of the MRMI scale reflects the patient's walking, bed-chair transfer, going up and down stairs, bathing items, and controlling urination; the lower limb-related part of the Berg scale reflects the patient's sitting to standing, independent standing, standing to sitting, bed-chair transfer, alternating steps with both feet, standing forward and backward with both feet, and standing on one leg items.

[0024] Further, the step S2 is specifically as follows:

[0025] S201: extracting a spatiotemporal skeleton graph from video data using a joint point extraction module;

[0026] S202: Extract scale features from text data using an encoding and normalization module.

[0027] Furthermore, the step S201 includes the following steps:

[0028] S2011: Acquire human skeleton information in the video image frame by frame;

[0029] The human skeleton information includes two-dimensional coordinate point information or three-dimensional coordinate point information of key nodes of the human skeleton and confidence N=[n x ,n y ,n d ], where (n x ,n y ) is the spatial coordinate of the skeleton node, n d The confidence level during skeleton detection;

[0030] S2012: Construct a spatiotemporal skeleton graph G = (N, L) consisting of a set of nodes (N) and a set of connection relationships (L); where N = {N ks |k=1,…,K;s=1,…,S}, the number of frames is K frames, and the number of key skeleton nodes is S; there are two types of connection relationship sets. One type of connection relationship set is the natural connection of skeleton nodes in the same frame. This connection relationship can be expressed as L c = {N ki N kj |(i,j)∈S}, another set of connection relations is the connection of the same node on consecutive frames, which can be expressed as L e = {N ki N(k+1)i}.

[0031] Furthermore, the step S202 includes the following steps:

[0032] S2021: Encode the text data list using one-hot encoding;

[0033] S2022: Normalize the encoding to obtain scale features.

[0034] Furthermore, according to the movement conditions, the Brunnstrom scale divides the limb dysfunction of the upper and lower limbs of stroke patients into stage I, stage II, stage III, stage IV, stage V, and stage VI, and the six stages mark the decrease of the degree of limb motor dysfunction in sequence; the output of the corresponding classifier is 6 dimensions, and the output value corresponds to the probability value of the Brunnstrom scale level of limb dysfunction. Usually, the level with the largest probability value is output as the limb dysfunction assessment result.

[0035] Furthermore, the loss function for the network training of the prediction module described in step S6 is a cross entropy loss function.

[0036] The beneficial effects of the present invention are as follows: a system and method for evaluating limb dysfunction in stroke patients are provided, a scale feature set for several patients is constructed using a KNN network based on the scale text data of the doctor's subjective judgment, and the limb node features and image features of the image data are used to eliminate the errors of subjective judgment, and the accuracy of the features corresponding to the image and the accuracy of the scale features finally screened are guaranteed through the cross-attention mechanism and the recommendation system, thereby achieving accurate prediction of the Brunnstrom scale rating corresponding to the limb dysfunction of stroke patients, assisting doctors in formulating subsequent rehabilitation treatment plans for patients, avoiding the interference of doctors' subjective cognition on the judgment results, and enriching the dimensional information of the features. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose and technical solution of the present invention more clear, the present invention provides the following drawings for explanation:

[0038] Figure 1 This is an architecture diagram of a limb dysfunction assessment system for stroke patients in Example 1 of the present invention;

[0039] Figure 2 This is a flow chart of a method for evaluating limb dysfunction in stroke patients in Example 2 of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose and technical solution of the present invention more clearly understood, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0041] Embodiment 1: It is now necessary to accurately and conveniently evaluate the recovery effect of stroke patients to guide the later rehabilitation training plan. In order to avoid the judgment bias caused by the subjective cognition of different doctors and avoid the complicated physical examination process, this embodiment provides "a limb dysfunction evaluation system for stroke patients".

[0042] Combination Figure 1 The invention comprises a data acquisition module (1), a data storage module (2), a data preprocessing module (3), a feature extraction module (4), a feature fusion module (5), a prediction module (6) and an output module (7); the data acquisition module (1) is used to acquire video data captured by a camera and text data of a patient scale; the data storage module (2) comprises a video data cache unit (21), a text data cache unit (22) and a feature storage unit (23) for storing data and features; the data preprocessing module (3) comprises a joint point extraction module (31) and an encoding and normalization module (32) for preprocessing video data and text data; the feature fusion module (5) comprises a KNN network (41), a GCN network (42) and a C3D network (43) for extracting features and constructing a feature space; the prediction module (6) comprises a recommendation system (61) and a classifier (62) for feature screening and classification; the output module (7) is an output interface for converting limb dysfunction data into a feature space. The evaluation result is fed back to the user; the text data cache unit (22) is respectively connected to the data acquisition module (1) and the encoding and normalization module (32); the video data cache unit (21) is respectively connected to the data acquisition module (1), the joint point extraction module (31) and the input of the C3D network (42); the feature storage unit (23) is connected to the feature extraction module (4); the input and output of the KNN network (41) are respectively connected to the output of the encoding and normalization module (32) and the input of the recommendation system (61); the input and output of the GCN network (42) are respectively connected to the output of the joint point extraction module (31) and the input of the feature fusion module (5); the output of the C3D network (42) is connected to the input of the feature fusion module (5); the output of the feature fusion module (5) is connected to the input of the recommendation system (61); the output of the recommendation system (61) is connected to the input of the classifier (62); and the output of the classifier (62) is connected to the output module (7).

[0043] Furthermore, the recommendation system (61) is a reinforcement learning network, whose state is the output of the feature fusion module (5), whose action space is the output of the KNN network (41), and whose reward is the correct percentage of the limb dysfunction assessment results output by the classifier (62).

[0044] Furthermore, the camera is a Kinect device; the joint extraction module is a Kinect software development tool, which is used to obtain the positions and connection relationships of human joint nodes in the image.

[0045] Furthermore, the feature fusion module (5) is a cross-attention mechanism for focusing on image features and limb node features.

[0046] Furthermore, the classifier (62) is composed of a fully connected network and a Softmax layer in series.

[0047] Furthermore, the number of the feature extraction module (4), the feature fusion module (5), and the prediction module (6) is 2, which respectively evaluate the limb dysfunction of the patient's upper limbs and lower limbs.

[0048] When working, the collected image data and text data are input into the data collection module (1) and the data storage module (2) of the system as input, and then the image data and text data are respectively subjected to preprocessing such as extraction of spatiotemporal skeleton graphs, encoding, and normalization by the preprocessing module (3), and then the features are respectively extracted by the feature extraction module (4), and then the node features and image features of the image data are focused by the feature fusion module (5), and finally the limb dysfunction of the stroke patient is evaluated by the prediction module (6), and the feedback is given to the doctor through the output module (7) to assist the doctor in specifying a rehabilitation plan.

[0049] Example 2: In combination with Example 1, it is now necessary to accurately and conveniently evaluate the recovery effect of stroke patients to guide the later rehabilitation training plan. In order to avoid the judgment bias caused by the subjective cognition of different doctors and avoid the complicated physical examination process, this example provides "a method for evaluating limb dysfunction in stroke patients".

[0050] Combination Figure 2 , including the following steps:

[0051] Step 1: Screen out the commonly used patient scales in clinical practice - ADL scale, MRMI scale for mobility assessment, Berg scale, and Brunnstrom scale, and store them in a list format as text data; and use the Kinect device to collect motion videos of stroke patients and store them frame by frame as video data.

[0052] Step 2: The data collection module (1) collects text data and video data, and stores them in corresponding positions of the storage module (2).

[0053] Step 3: The data preprocessing module (3) preprocesses the video data and text data to obtain the spatiotemporal skeleton graph and scale features. Specifically:

[0054] 1) extracting a spatiotemporal skeleton graph from the video data using a joint point extraction module (31);

[0055] 2) Using the encoding and normalization module (32) to extract scale features from text data.

[0056] Furthermore, the step 1) comprises the following steps:

[0057] 1-1) Obtaining human skeleton information in video images frame by frame;

[0058] The human skeleton information includes two-dimensional coordinate point information or three-dimensional coordinate point information of key nodes of the human skeleton and confidence N=[n x ,n y ,n d ], where (n x ,n y ) is the spatial coordinate of the skeleton node, n d The confidence level during skeleton detection;

[0059] 1-2) Construct a spatiotemporal skeleton graph G = (N, L) consisting of a set of nodes (N) and a set of connection relationships (L); where N = {N ks |k=1,…,K;s=1,…,S}, the number of frames is K frames, and the number of key skeleton nodes is S; there are two types of connection relationship sets. One type of connection relationship set is the natural connection of skeleton nodes in the same frame. This connection relationship can be expressed as L c = {N ki N kj |(i,j)∈S}, another set of connection relations is the connection of the same node on consecutive frames, which can be expressed as L e = {N ki N (k+1)i}.

[0060] Furthermore, the step 2) comprises the following steps:

[0061] 2-1) Encode the text data list using one-hot encoding;

[0062] 2-2) The encoding is normalized to obtain the scale feature. The normalization is performed by using MaxMinScaler to scale to the interval [0,1].

[0063] Step 4: Use the KNN network (41) to select K scale features that are closest to the patient's scale features in the historical data and construct a scale feature set.

[0064] Step 5: Use the C3D network (43) to extract image features of the video data, and use the GCN network (42) to extract limb node features of the spatiotemporal skeleton graph.

[0065] Step six: Use the feature fusion module (5) to fuse the image features and the limb node features to obtain the limb node fusion features.

[0066] Step 7: Use the obstacle level classification of the Brunnstrom scale as labels and use historical data to train the network in the prediction module (6).

[0067] The output of the classifier (62) in the prediction module (6) is 6-dimensional, and the output value corresponds to the probability value of the level of the Brunnstrom scale of limb dysfunction.

[0068] The loss function for the network training of the prediction module (6) is a cross entropy loss function.

[0069] During training, the network in the prediction module can also be trained together with the KNN network (41), the GCN network (42) and the C3D network (43).

[0070] Step 8: Input the limb node fusion feature into the trained recommendation system (61), and use the recommendation system (61) to provide a scale feature that is closest to the limb node fusion feature as the scale feature expression.

[0071] Step nine: Input the scale feature expression into the trained classifier (62) to evaluate the severity of the patient's limb dysfunction. The Brunnstrom scale level with the largest probability value is used as the limb dysfunction evaluation result, and the output module (7) is used to feed back to the doctor to assist the doctor in specifying a rehabilitation plan.

[0072] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A limb dysfunction assessment system for stroke patients, characterized in that: The invention is composed of a data acquisition module (1), a data storage module (2), a data preprocessing module (3), a feature extraction module (4), a feature fusion module (5), a prediction module (6) and an output module (7); the data acquisition module (1) is used to acquire video data captured by a camera and text data of a patient scale; the data storage module (2) comprises a video data cache unit (21), a text data cache unit (22) and a feature storage unit (23) for storing data and features; the data preprocessing module (3) comprises a joint point extraction module (31) and an encoding and normalization module (32) for preprocessing video data and text data; the feature fusion module (5) comprises a K-Nearest Neighbor (KNN) network (41), a Graph Convolutional Network (GCN) network (42) and a C3D (3D convolution, C3D) network (43) for extracting features and constructing a feature space; The prediction module (6) comprises a recommendation system (61) and a classifier (62) for feature screening and classification; the output module (7) is an output interface for feeding back the limb dysfunction assessment result to the user; the text data cache unit (22) is respectively connected to the data acquisition module (1) and the encoding and normalization module (32); the video data cache unit (21) is respectively connected to the data acquisition module (1), the joint point extraction module (31), and the input of the C3D network (42); the feature storage unit (23) is connected to the feature extraction module (4); the KNN network The input and output of (41) are respectively connected to the output of the encoding and normalization module (32) and the input of the recommendation system (61); the input and output of the GCN network (42) are respectively connected to the output of the joint point extraction module (31) and the input of the feature fusion module (5); the output of the C3D network (42) is connected to the input of the feature fusion module (5); the output of the feature fusion module (5) is connected to the input of the recommendation system (61); the output of the recommendation system (61) is connected to the input of the classifier (62); and the output of the classifier (62) is connected to the output module (7).

2. The limb dysfunction assessment system for stroke patients according to claim 1, characterized in that: The recommendation system (61) is a reinforcement learning network, whose state is the output of the feature fusion module (5), whose action space is the output of the KNN network (41), and whose reward is the correct percentage of the limb dysfunction assessment results output by the classifier (62).

3. The limb dysfunction assessment system for stroke patients according to claim 1, characterized in that: The camera is a Kinect device; the joint extraction module is a Kinect software development tool, which is used to obtain the positions and connection relationships of human joint nodes in the image.

4. The limb dysfunction assessment system for stroke patients according to claim 1, characterized in that: The feature fusion module (5) is a cross-attention mechanism, which is used to focus on image features and limb node features.

5. The limb dysfunction assessment system for stroke patients according to claim 1, characterized in that: The classifier (62) is composed of a fully connected network and a Softmax layer in series.

6. The limb dysfunction assessment system for stroke patients according to claim 1, characterized in that: The number of the feature extraction module (4), the feature fusion module (5) and the prediction module (6) is 2, and the limb dysfunction of the patient's upper limb and lower limb are evaluated respectively.

7. A method for evaluating limb dysfunction in stroke patients as claimed in any one of claims 1 to 6, characterized in that: The following steps are included: S1: inputting the video data of the patient's movement collected by the camera and the text data of the patient's scale into the data acquisition module (1), and storing them in the corresponding position of the storage module (2); S2: Data preprocessing module (3) preprocesses the video data and text data to obtain spatiotemporal skeleton graph and scale features respectively; S3: Use the KNN network (41) to select several scale features that are closest to the patient's scale features in the historical data and construct a scale feature set; S4: using the C3D network (43) to extract image features of the video data, and using the GCN network (42) to extract limb node features of the spatiotemporal skeleton graph; S5: using the feature fusion module (5) to fuse the image features and the limb node features to obtain the limb node fusion features; S6: Using the Brunnstrom scale obstacle level classification as labels, the network in the prediction module (6) is trained using historical data; S7: input the limb node fusion feature into the trained recommendation system (61), and use the recommendation system (61) to give a scale feature that is closest to the limb node fusion feature as the scale feature expression; S8: Input the scale feature expression into the trained classifier (62) to evaluate the severity of the patient's limb dysfunction.

8. The method for evaluating limb dysfunction in stroke patients according to claim 7, characterized in that: The text data of the scale includes ADL scale, MRMI scale and Berg scale, which are stored in list format; wherein, the upper limb related part of the ADL scale reflects the patient's eating, bed and chair transfer, going to the toilet, grooming, dressing and bathing items; the lower limb related part of the ADL scale reflects the patient's walking, going up and down stairs, bed and chair transfer, going to the toilet, controlling bowel movements, bathing and controlling urination items; the upper limb related part of the MRMI scale reflects the patient's turning over in bed, bed and chair transfer, grooming, dressing and bathing items; the lower limb related part of the MRMI scale reflects the patient's walking, bed and chair transfer, going up and down stairs, bathing items and controlling urination; the lower limb related part of the Berg scale reflects the patient's sitting to standing, independent standing, standing to sitting, bed and chair transfer, alternating steps with both feet, standing forward and backward with both feet, and standing on one leg items.

9. The method for evaluating limb dysfunction in stroke patients according to claim 7, characterized in that: The step S2 is specifically as follows: S201: extracting a spatiotemporal skeleton graph from the video data using a joint point extraction module (31); S202: extracting scale features from text data using the encoding and normalization module (32); Wherein, the step S201 includes the following steps: S2011: acquiring human skeleton information in the video image frame by frame; the human skeleton information includes coordinate point information and confidence of key nodes of the human skeleton; S2012: Construct a spatiotemporal skeleton graph consisting of a set of nodes and a set of connection relationships; Wherein, the step S202 comprises the following steps: S2021: Encode the text data list using one-hot encoding; S2022: Normalize the encoding to obtain scale features.

10. The method for evaluating limb dysfunction in stroke patients according to claim 7, characterized in that: The Brunnstrom scale described in step S6 classifies the limb dysfunction of the upper limbs and lower limbs of stroke patients into stage I, stage II, stage III, stage IV, stage V, and stage VI, which correspond to the six dimensions output by the classifier (62), and the output value corresponds to the probability value of the level of the Brunnstrom scale of limb dysfunction; the loss function of the network training of the prediction module (6) is a cross entropy loss function.

Images