A stroke patient upper limb movement evaluation method and upper limb movement evaluation system

Abstract

The application discloses a kind of upper limb movement evaluation method and upper limb movement evaluation system of stroke patient, former includes: the video when the two sides of stroke patient upper limb successively complete predetermined movement process is collected;The target that the two sides of upper limb complete predetermined movement process is, the movement track formed along the vertical midline of stroke patient is axisymmetric;Extract skeletal key point data in video;Skeletal key point data characterizes the position information of each skeletal key point of stroke patient upper limb at corresponding moment;Based on the feature difference of two sides of skeletal key point data in predetermined movement process, determine the NIHSS score of stroke patient, obtain upper limb movement evaluation result.The present application utilizes computer vision technology, can realize to upper limb movement objective, automatic detection, improve evaluation objectivity and standardization degree, can real-time detection and analysis, improve evaluation efficiency, can provide timely data support for doctor, help to formulate individualized rehabilitation plan and adjust treatment scheme.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical rehabilitation, and particularly relates to a method and system for evaluating upper limb movement of a stroke patient. BACKGROUND

[0002] Stroke is a common neurological disease, and the resulting upper limb motor dysfunction is a major problem in the rehabilitation process of patients. Rehabilitation assessment of stroke patients is an important problem in the medical field. Stroke patients often face problems of impaired upper limb motor function, so accurate and objective assessment of their upper limb motor function is of great significance for developing personalized rehabilitation programs and improving rehabilitation outcomes.

[0003] Stroke patient upper limb movement scoring techniques usually involve the use of a series of assessment tools and methods to determine the degree of upper limb function and rehabilitation progress of patients. The closest prior art includes: 1) Fugl-Meyer Assessment (FMA): This is a commonly used method for assessing upper limb function in stroke patients, including muscle strength, motor control, balance and coordination. 2) Unified Parkinson's Disease Rating Scale (UPDRS): Although primarily used to assess symptoms and function in patients with Parkinson's disease, it can also be used to partially assess upper limb function in patients with motor dysfunction. 3) National Institutes of Health Stroke Scale (NIHSS): Although it is primarily used to assess the severity of clinical symptoms in stroke patients, it can also be used to assess upper limb movement in patients with impaired motor function. These techniques are common clinical tools for assessing upper limb motor function in stroke patients and can help rehabilitation professionals understand the rehabilitation progress of patients and develop personalized rehabilitation plans.

[0004] In addition to the above-mentioned upper limb motor function assessment methods, traditional methods such as Brunnstrom motor function assessment and ARAT assessment, etc. Although these methods have certain clinical application value, they have the following defects:

[0005] Labor intensive: Traditional upper limb function assessment usually requires the manual operation of professional medical personnel, including face-to-face observation of patients, physical tests, and recording of assessment results. Such an assessment process usually requires a large amount of human resources, especially in cases where medical resources are limited, which can limit patients' access to timely and comprehensive assessment services.

[0006] Inaccurate: Since traditional methods mainly rely on manual observation and manual testing, the evaluation results are often affected by the subjective factors of medical personnel, and there is a certain subjectivity and uncertainty. In addition, the professional level and experience of medical personnel will also affect the accuracy of the evaluation results, so there may be certain evaluation differences between different doctors.

[0007] Non-standard: In traditional methods, due to the lack of unified evaluation standards and processes, different medical institutions or doctors may use different evaluation methods and standards, resulting in non-standard and incomparable evaluation results. This not only affects the tracking and comparison of patients' treatment and rehabilitation progress, but also may lead to inconsistent and misunderstanding of evaluation results.

[0008] Time-consuming: Traditional methods often require a long time to complete a complete evaluation process, including waiting time, testing time and recording time, etc. This may be a burden for medical institutions and doctors, especially in a busy clinical work environment, the consumption of time may limit the doctors to provide comprehensive and timely evaluation services to patients.

[0009] As can be seen, the traditional method has many shortcomings in evaluating the upper limb function of stroke patients, such as consuming manpower, complex operation, long time-consuming, inaccuracy, non-standard, etc. Therefore, it is of great significance to find a more objective, accurate and efficient evaluation method for the rehabilitation treatment and functional recovery of stroke patients. SUMMARY

[0010] In order to solve the above problems existing in the prior art, the present application provides a method for evaluating the upper limb movement of a stroke patient and an upper limb movement evaluation system. The technical problem to be solved by the present application is solved by the following technical scheme:

[0011] In a first aspect, the present application provides a method for evaluating the upper limb movement of a stroke patient, which comprises:

[0012] Collecting videos of the stroke patient's bilateral upper limbs completing a predetermined movement process in sequence; wherein the goal of the bilateral upper limbs completing the predetermined movement process is that the movement trajectories formed by the two sides are axisymmetric along the vertical midline of the stroke patient;

[0013] Extracting bone key point data in the video; wherein the bone key point data represents the position information of each bone key point of the stroke patient's upper limb at the corresponding time;

[0014] Based on the feature difference of the bilateral upper limbs in the predetermined movement process in the bone key point data, determining the NIHSS score of the stroke patient, and obtaining the upper limb movement evaluation result.

[0015] In an embodiment of the present application, the predetermined movement process of the single upper limb includes:

[0016] The hand is lifted from the natural position on the side of the body, and experiences a plurality of predetermined positions until the movement process of touching the tip of the nose.

[0017] In an embodiment of the present application, the skeleton key point data in the video is extracted, and a MediaPipe library is used to realize.

[0018] In an embodiment of the present application, the skeleton key point data in the video is extracted, and a preset network based on a ResNet50 model is used to realize; wherein the preset network is obtained by training using a picture data set with node annotation information.

[0019] The preset network includes a preset convolutional layer, a feature extraction module, a two-dimensional global average pooling layer, a Dropout layer and a Dense layer connected in sequence; wherein the preset convolutional layer is used to adjust the dimension of the input data, the preset convolutional layer is a two-dimensional convolutional layer, uses a 1x1 Kernel kernel, and uses a LeakyReLU activation function; the feature extraction module is composed of the convolutional layer in the ResNet50 model; the two-dimensional global average pooling layer is used to aggregate features; the Dropout layer and the Dense layer are fully connected layers, the Dropout layer is used to reduce overfitting; and the Dense layer is used as the output layer of the preset network.

[0020] In an embodiment of the present application, each skeleton key point of the upper limb of the stroke patient includes a predetermined point on the left hand, the left elbow, the left shoulder, the right hand, the right elbow, and the right shoulder.

[0021] In an embodiment of the present application, based on the feature difference between the two upper limbs in the predetermined movement process in the skeleton key point data, the NIHSS score of the stroke patient is determined, and an upper limb movement evaluation result is obtained, including:

[0022] Using the skeleton key point data, the effective movement data sequence corresponding to each side of the upper limb during the predetermined movement process of the stroke patient is determined;

[0023] The effective movement data sequence of one side of the upper limb is mirror flipped using a flip flipping technology to obtain the contralateral effective movement data sequence;

[0024] The effective movement data sequence of the other side of the upper limb which is not mirror flipped and the contralateral effective movement data sequence are compared in sequence similarity using a DTW algorithm to obtain a difference degree measurement value;

[0025] determine the NIHSS score according to the difference degree measurement value, and obtain the upper limb movement evaluation result of the stroke patient.

[0026] In an embodiment of the present application, the effective movement data sequence corresponding to each side of the upper limb of the stroke patient during the predetermined movement process is determined by using the skeleton key point data, including:

[0027] Based on the skeleton key point data determined in each frame of the video, the effective video segment corresponding to the stroke patient only completing the predetermined movement process of the left upper limb and the effective video segment corresponding to the stroke patient only completing the predetermined movement process of the right upper limb are determined.

[0028] For each side of the effective video segment, all skeleton key point data in each frame are arranged in order, and a plurality of vectors are obtained according to the skeleton key point data before and after the arrangement. The included angle between each two vectors is calculated to obtain the included angle sequence corresponding to the frame. The included angle sequences obtained from all frames are connected in the order of frames to obtain the effective movement data sequence of the corresponding side.

[0029] In an embodiment of the present application, the NIHSS score of the stroke patient is determined based on the feature difference of the two sides of the upper limb in the predetermined movement process, and the upper limb movement evaluation result is obtained, including:

[0030] The time sequence corresponding to the skeleton key point data extracted from the video is input into the pre-trained LSTM model to obtain the model difference degree measurement of the upper limb movement of the stroke patient.

[0031] The NIHSS score is determined based on the model difference degree measurement to obtain the upper limb movement evaluation result.

[0032] The LSTM model is trained by using the sample time sequence corresponding to the skeleton key point data extracted from the sample video of the two sides of the upper limb of different individuals during the predetermined movement process, and the corresponding label. The label of different sample time sequences represents the different model difference degree measurement results of the two sides of the upper limb of the corresponding individual. The LSTM model includes three LSTM layers and an LSTM output layer, and the three LSTM layers have an increasing number of LSTM units and then a decreasing number of LSTM units. The number of units of the LSTM output layer is dynamically set.

[0033] In an embodiment of the present application,

[0034] In a second aspect, the embodiments of the present application provide an upper limb movement evaluation system for stroke patients, including a user end, a server end and a database end, wherein:

[0035] The user end is used for collecting videos of the stroke patient when the two upper limbs complete the predetermined movement process in turn, and extracting bone key point data in the videos; wherein the target of the two upper limbs completing the predetermined movement process is that the movement trajectories formed on the two sides are axisymmetric along the vertical midline of the stroke patient; the bone key point data represents position information of each bone key point of the upper limb of the stroke patient at the corresponding moment;

[0036] The server end is used for receiving the bone key point data sent by the user end, determining the NIHSS score of the stroke patient based on feature differences of the two upper limbs in the predetermined movement process in the bone key point data, obtaining an upper limb movement evaluation result, and sending the upper limb movement evaluation result to the user end and the database end.

[0037] The database end is used for storing the upper limb movement evaluation result of each stroke patient.

[0038] In an embodiment of the present application, the upper limb movement evaluation system is provided with a historical score acquisition interface, and the user end acquires multiple NIHSS scores of the stroke patient from the database end through the service end by using the historical score acquisition interface, so as to analyze the NIHSS score trend.

[0039] The present application has the following beneficial effects:

[0040] The upper limb movement evaluation method for stroke patients provided by the embodiment of the present application first collects videos of the stroke patient when the two upper limbs complete the predetermined movement process in turn; wherein the target of the two upper limbs completing the predetermined movement process is that the movement trajectories formed on the two sides are axisymmetric along the vertical midline of the stroke patient; then bone key point data in the videos is extracted; wherein the bone key point data represents position information of each bone key point of the upper limb of the stroke patient at the corresponding moment; finally, the NIHSS score of the stroke patient is determined based on feature differences of the two upper limbs in the predetermined movement process in the bone key point data, and an upper limb movement evaluation result is obtained.

[0041] Traditional upper limb movement assessment usually relies on the subjective observation and manual testing of medical personnel, thus there are problems of subjectivity and inaccuracy. The application can realize objective and automatic detection of upper limb movement by using computer vision technology, thereby improving the objectivity and standardization of the evaluation. Traditional upper limb movement assessment often requires a large amount of time and human resources, which may be a burden for medical institutions and doctors, especially in a busy clinical work environment. The application can realize real-time detection and analysis of upper limb movement, greatly improving the efficiency and time effect of the evaluation, which helps to discover the movement changes of patients in time and carry out targeted rehabilitation intervention. The rehabilitation process of stroke patients is usually a continuous process that requires long-term tracking and evaluation. The application can realize continuous monitoring and tracking of the upper limb movement of patients, can record the movement state and change trend of patients in real time, and provide timely data support for doctors, which helps to develop personalized rehabilitation plans and adjust treatment programs.

[0042] Further, the embodiment of the application can use the MediaPipe library or the preset network based on the ResNet50 model to realize the extraction of the skeletal key points, and can use the DTW algorithm or the LSTM model to realize the NIHSS score, so as to obtain the upper limb movement evaluation result. The application realizes real-time monitoring, analysis and evaluation of the upper limb movement of patients, not only improves the accuracy and objectivity of the evaluation, but also greatly simplifies the operation process and improves the evaluation efficiency. It can better serve the treatment and rehabilitation needs of stroke patients.

[0043] The upper limb movement evaluation system for stroke patients provided by the embodiment of the application includes a user end, a server end and a database end, uses computer vision technology to provide upper limb movement evaluation for stroke patients, can provide more comprehensive and objective patient information for medical personnel, and helps to develop personalized rehabilitation treatment programs and track rehabilitation progress. The application adopts a variety of technical solutions, combines computer vision, deep learning, database and other technologies, realizes real-time monitoring and analysis of the upper limb movement of stroke patients. This not only improves the efficiency and time effect of the evaluation, but also helps to discover the movement changes of patients in time and carry out targeted rehabilitation intervention. At the same time, the application also has good scalability and customizability, which can be improved and optimized according to specific needs. It is believed that this technology will have a positive impact on the rehabilitation treatment and health management of stroke patients. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A flowchart of a method for evaluating the upper limb movement of a stroke patient is provided by the embodiment of the application;

[0045] Figure 2 A schematic diagram of the principle of training a preset network is provided;

[0046] Figure 3 is a structural schematic diagram of the preset network described in the embodiment of the present application;

[0047] Fig. 4(a) and Fig. 4(b) are respectively the model accuracy and model loss results of the preset network training process in the embodiment of the present application;

[0048] Figure 5 is a hand longitudinal coordinate separation diagram in the embodiment of the present application;

[0049] Figure 6 is a processing process schematic diagram of the DTW algorithm in the embodiment of the present application;

[0050] Figure 7 is a process schematic diagram of collecting video until obtaining the NIHSS score in the embodiment of the present application, taking the MediaPipe library combined with the DTW algorithm as an example;

[0051] Figure 8 is an example of the difference degree measurement value, taking the MediaPipe library combined with the DTW algorithm as an example in the embodiment of the present application;

[0052] Figure 9 is an example of the NIHSS score obtained by different stroke patients in the embodiment of the present application;

[0053] Figure 10 is a structural diagram of the LSTM network;

[0054] Figure 11 is a structural schematic diagram of an upper limb movement evaluation system for a stroke patient in the embodiment of the present application;

[0055] Figure 12 is a structural schematic diagram of an upper limb movement evaluation system, taking the MediaPipe library combined with the DTW algorithm as an example in the embodiment of the present application;

[0056] Figure 13 is a flowchart of starting to collect video until obtaining the upper limb movement evaluation result in the system example of the embodiment of the present application;

[0057] Fig. 14(a) and Fig. 14(b) are effect diagrams of the historical score curve in the embodiment example of the present application. DETAILED DESCRIPTION

[0058] The present application will be further described in detail below in conjunction with specific embodiments, but the implementation manner of the present application is not limited thereto.

[0059] With the continuous development of medical technology, the rehabilitation treatment effect of stroke patients has been significantly improved. However, how to accurately and objectively evaluate the rehabilitation progress of patients has been an important problem faced by the medical field. Since the traditional evaluation method for the upper limb function of stroke patients often depends on the subjective judgment and experience of doctors, there are certain limitations and errors, resulting in many defects. The present application considers using computer vision technology to realize a more objective, accurate and efficient evaluation method. Specifically, based on the scoring method of NIHSS, an upper limb movement evaluation method and system for stroke patients is proposed.

[0060] For ease of understanding, first, the NIHSS is briefly introduced.

[0061] The National Institutes of Health Stroke Scale (NIHSS) is a standardized tool widely used to assess the severity of clinical symptoms in stroke patients. It consists of a series of items that cover a variety of symptoms that stroke patients may experience, including level of consciousness, eye movement, limb movement, facial muscle activity, language ability, etc.

[0062] The main purpose of NIHSS is to evaluate the severity of symptoms in stroke patients to guide treatment decisions, but its limb movement items can also be used to evaluate the upper limb function of patients. These limb movement items include evaluating the strength, coordination and degree of impairment of the upper limb function of patients. In clinical practice, doctors often use the limb movement items to preliminarily evaluate the upper limb function of patients.

[0063] In NIHSS, the limb movement items usually involve the ability of patients to perform specific actions, such as grip strength, finger extension, etc. Doctors will observe whether patients can complete these actions according to the instructions, and evaluate the upper limb function according to the ability and flexibility of patients to complete the actions. Through the evaluation of these items, doctors can preliminarily understand the degree of impairment of the upper limb function of patients, and accordingly develop subsequent treatment and rehabilitation plans. The scoring criteria can be seen in Table 1.

[0064] Table 1

[0065]

[0066] Next, the embodiment scheme of the present application is specifically described.

[0067] In the first aspect, the embodiment of the present application provides an upper limb movement evaluation method for stroke patients, as shown in Figure 1 The method can include the following steps:

[0068] S1, collecting videos of the stroke patient's two sides upper limbs completing the predetermined movement process in sequence;

[0069] The video can be captured by using an existing camera.

[0070] In an optional embodiment, the video of the stroke patient performing the predetermined movement process on both sides of the upper limbs is captured, and the video stream is read by using the OpenCV library to perform frame-level processing and analysis on the video. The OpenCV provides rich video processing functions, including video stream acquisition, saving, playing, etc., and has good cross-platform and compatibility, and can conveniently realize the operation on the video in various environments.

[0071] Since the stroke patient can have movement disorder of one side of the upper limb, when the video of the stroke patient performing the predetermined movement process on both sides of the upper limbs is captured, the target of the predetermined movement process of the upper limbs on both sides is that the movement trajectories formed on both sides are axisymmetric along the vertical midline of the stroke patient. That is, in the absence of disorder, the movement trajectories of the upper limbs on both sides of the stroke patient are mirror-symmetric along the vertical midline of the body. Of course, the movement trajectories of the upper limbs on both sides of the stroke patient with upper limb disorder cannot be mirror-symmetric along the vertical midline of the body.

[0072] When the predetermined movement process is completed, the patient can first perform the action on one side and then perform the action on the other side, and the captured video includes a complete process of performing the movement process on both sides.

[0073] For example, the specific action can be lifting the single-arm from the side of the body to the top of the head, or lifting the single-arm from the side of the body to touch the ear lobe on the side, etc.

[0074] In an optional embodiment, the predetermined movement process of the single upper limb includes:

[0075] The hand is lifted from the natural position falling on the side of the body, and experiences a movement process of multiple predetermined positions until the nose tip is touched.

[0076] Specifically, the left side can be performed first, or the right side can be performed first, and the specific implementation is not limited, but it is required that when the stroke patient performs the predetermined movement process on each side, the trajectory on the side is completed along the multiple predetermined positions on the single side, and the distribution of the same predetermined position on both sides is axisymmetric along the body midline. For example, it is required that when the actions on both sides are performed, the body is in a vertical state, the predetermined starting point on each side is at the position where the hand naturally falls on the side of the body, the small arm is lifted to drive the large arm to move in the plane of the side of the body until the nose tip is touched, and the action is ended as the predetermined end point.

[0077] The embodiment selects the nose tip as a point located on the center line of the human body, which is convenient for patient positioning and comparison of actions on both sides. For ease of understanding and description, the predetermined movement process of lifting the hand to touch the nose tip is described as an example in the following.

[0078] Of course, the embodiment of the application can also artificially set a point located on the center line of the human body, such as the center of the head, as a predetermined end point, which is reasonable, and the specific implementation is not limited.

[0079] S2, extracting the skeletal key point data in the video;

[0080] It can be understood that the collected video contains the complete action process of the stroke patient when the upper limbs on both sides complete the predetermined movement process in sequence, such as lifting the left hand, touching the nose tip, then putting down the left hand, lifting the right hand, touching the nose tip, and then putting down the right hand, which actually contains multiple frames of video.

[0081] The embodiment of the application can extract the skeletal key point data in the video; wherein the skeletal key point data represents the position information of each skeletal key point of the upper limbs of the stroke patient at the corresponding time, and the position information mainly includes the coordinates of the skeletal key points in the horizontal direction and the coordinates in the vertical direction, which can be represented by x coordinates and y coordinates respectively.

[0082] Each skeletal key point represents some specific points in the upper limbs, such as, in an optional embodiment, the skeletal key points of the upper limbs of the stroke patient include the predetermined points of the left hand, the left elbow, the left shoulder, the right hand, the right elbow, and the right shoulder respectively.

[0083] In order to position more accurately, the predetermined point on the hand can be the tip of a specific finger, such as the tip of the thumb or the middle finger, the predetermined point of the elbow can be the center position of the elbow, the predetermined point of the shoulder can be the highest point of the shoulder, etc., but the positions of the predetermined points used by the left and right hands, elbows, and shoulders respectively need to be the same.

[0084] In an optional embodiment, the skeletal key point data in the video is extracted, and the skeletal key point data in the video is extracted by using the MediaPipe library.

[0085] The MediaPipe library is an open source library provided by Google, which can realize real-time detection and tracking of key points of the skeleton in the upper limb movement of a stroke patient. MediaPipe can identify the key points of the skeleton of the human body, including the key points of the skeleton of the arms and hands, through deep learning technology, and track their movement trajectories in real time. Generally, the MediaPipe library identifies more key points of the skeleton, including the key points of the skeleton of the whole body, and the key points of the skeleton of the upper limbs can be selected as needed. For example, the hand skeleton key point detection model in the MediaPipe library can effectively identify and extract the key points of the skeleton of the human body, such as shoulders, elbows, wrists, etc., from videos or images, and the accurate detection of these key points of the skeleton is the basis for subsequent motion analysis.

[0086] In an optional another embodiment, the key point data of the skeleton in the video is extracted, and a preset network based on a ResNet50 model is used to realize the extraction; wherein the preset network is obtained by training using a picture data set with joint annotation information; the picture data set comes from sample videos of the two sides of the upper limbs of an individual when performing a predetermined movement process, and the joint annotation information is the annotated key points of the skeleton;

[0087] Specifically, please refer to Figure 2 , Figure 2 The principle diagram for training the preset network; the preset network is designed by the embodiment of the application to improve the accuracy and robustness of the detection of the key points of the skeleton, and the preset network is a deep learning model based on a ResNet50 model. The ResNet50 model is a deep residual network, which effectively solves the gradient disappearance and representation bottleneck problem in the deep neural network by introducing a residual block, so that the network can be deeper and more complex, thereby extracting more rich feature information.

[0088] In the training process, a large amount of human pose estimation data set is collected in advance, including human images in different angles, different light conditions and different actions. These data come from different individuals, i.e. videos or images of healthy people and stroke patients when performing the same action. Specifically, the data in the sample videos collected when the two sides of the upper limbs complete a predetermined movement process are used as labels for supervised training in the training process;

[0089] The stochastic gradient descent (SGD) optimization algorithm is used in the training process, and appropriate learning rate and iteration number are set to make the preset network converge to the optimal solution.

[0090] Please refer to Figure 3 , Figure 3A structure diagram of a preset network in an embodiment of the present application.

[0091] The preset network comprises a preset convolutional layer, a feature extraction module, a two-dimensional global average pooling layer, a Dropout layer and a Dense layer connected in sequence; wherein the preset convolutional layer is used for adjusting the dimension of input data, the preset convolutional layer is a two-dimensional convolutional layer, adopts a 1*1 Kernel kernel and adopts a LeakyReLU activation function; the feature extraction module is constituted by using the convolutional layer in the ResNet50 model; the two-dimensional global average pooling layer is used for aggregating features; the Dropout layer and the Dense layer are fully connected layers, the Dropout layer is used for reducing overfitting; and the Dense layer serves as an output layer of the preset network.

[0092] The preset network in the embodiment of the present application is implemented based on the ResNet50 model, and setting include_top=False means that only the convolutional layer of the ResNet50 model is used, and the top fully connected layer (usually used for classification) is not included. By setting pretrained_model.trainable=True, the weights of the ResNet50 model are allowed to be fine-tuned in the subsequent training process. This can usually improve the performance of the model on a specific task. Before the feature extraction module, the embodiment of the present application adds a preset convolutional layer, which includes a custom Conv2D layer, a 1*1 Kernel kernel function, and a LeakyReLU activation function for the Conv2D layer. This can be regarded as a kind of preprocessing or feature conversion for the input data to better adapt to the input of the ResNet50 model.

[0093] The original ResNet50 model is designed for 224x224 pixel images, but the embodiment of the present application adjusts the input size to 128x128. This can reduce the amount of calculation and memory usage, but may also sacrifice some accuracy because a smaller input may not be able to capture all the details of the image. The custom Conv2D layer accepts an input of (128, 128, 1), which means that the model is designed for single-channel (grayscale) images. If color images are used in actual applications, the number of input channels needs to be adjusted accordingly.

[0094] After the ResNet50 model implements the feature extraction module, the embodiment of the present application only adds a two-dimensional global average pooling layer (denoted as a GlobalAveragePooling2D layer) to aggregate features, a Dropout layer to reduce overfitting, and a Dense layer for output. This simplified architecture can reduce the complexity and computational cost of the model.

[0095] The Dense layer of the embodiment of the present application can have multiple output nodes, such as 12 output nodes. For example, 6 bone key points (x, y) 12 numbers. According to the specific task, the number of output nodes may need to be adjusted.

[0096] The preset model obtained by training in the embodiment of the present application can accurately identify and understand the human body posture in the image or video, and can accurately identify the bone key points of the upper limbs of the human body, including hands, arms, elbows, shoulders, etc. In the process of real-time monitoring and analyzing the movement of the upper limbs of the patient, the model can be used to detect the bone key points of the human body in the video stream, so as to obtain more accurate movement trajectory and posture information, and can capture the subtle movements of the upper limbs of the patient, such as the lifting of the arm, so as to provide rich movement data. This helps to more accurately evaluate the movement function state of the patient and develop a personalized rehabilitation treatment plan for the patient.

[0097] In the field of machine learning and deep learning, the evaluation and optimization of the model is a core link. For the preset network, the correct rate and loss function on the test set are visualized, which can more intuitively understand the performance.

[0098] After many experiments and optimizations, the accuracy of the preset network of the embodiment of the present application on the test set reaches 92%. This achievement not only exceeds most similar models, but also proves that the efforts of the embodiment of the present application in model design, parameter adjustment, and data preprocessing have achieved remarkable results.

[0099] Regarding the loss function of the model. The loss function is the key to the optimization of the machine learning model, which measures the gap between the model's prediction results and the actual results. During the training process, the goal is to continuously reduce this gap, so that the model's prediction results become more and more close to the true value.

[0100] As can be seen from FIG. 4(a) and FIG. 4(b), with the training, the loss function value of the preset network gradually decreases and finally stabilizes at a lower level. This shows that the preset network gradually learns to extract useful information from data during the training process and continuously optimizes its prediction ability. At the same time, this also proves the effectiveness of various optimization algorithms and regularization techniques used in the training process of the embodiment of the present application. FIG. 4(a) represents the model accuracy, FIG. 4(b) represents the model loss, and the blue curve represents the training set effect, and the orange curve represents the test set effect.

[0101] S3, based on the feature difference of the two upper limbs in the preset movement process in the bone key point data, determining the NIHSS score of the stroke patient to obtain an upper limb movement evaluation result.

[0102] 1) In an optional implementation, S3 can include the following steps:

[0103] Step A1, determining the effective motion data sequence corresponding to each side of the upper limb of the stroke patient during the predetermined motion process of each side of the upper limb of the stroke patient by using the skeleton key point data;

[0104] Since the video collected is the complete video of the stroke patient completing the predetermined motion process in sequence, the video segment required by the single side from the predetermined starting point to the predetermined key point is required, while some action segments after reaching the predetermined end point and some action segments before the starting point are not required, and therefore, the single side of the truly required video segment needs to be obtained from the complete video collected.

[0105] In an optional embodiment, step A1 can include the following steps:

[0106] Step A11, determining the effective video segment corresponding to the stroke patient only completing the predetermined motion process of the left upper limb and the effective video segment corresponding to the stroke patient only completing the predetermined motion process of the right upper limb based on the skeleton key point data determined in each frame of the video;

[0107] Step A12, for each side of the effective video segment, arranging all the skeleton key point data in each frame in sequence, obtaining a plurality of vectors according to the skeleton key point data before and after the sequence, calculating the included angle between each two vectors to obtain the included angle sequence corresponding to the frame, and connecting the included angle sequences obtained from all the frames in the order of the frames to obtain the effective motion data sequence of the corresponding side.

[0108] Specifically, the video stream is read frame by frame through the OpenCV library, and the position of the hand and the nose of the stroke patient and the motion relationship between the hand and the nose in each frame can be recognized by using the MediaPipe library and the like. For the position information of the hand and the nose in each frame, the association and matching of the hand and the nose can be performed by using the least value approximation algorithm, so as to determine whether the action of the patient touching the nose with the hand occurs in each frame. According to the occurrence of the action of touching the nose with the hand, the continuous action in the video stream can be segmented into a single hand-nose touching action sequence. This can be achieved by detecting the interval or change between the continuous actions, for example, judging the start and end of the action according to the change trend of the position of the hand and the nose or the speed of the action.

[0109] Figure 5 The curve in the above formula is the ordinate of the hand, and when the patient touches the nose, the hand coordinate will decrease, such as Figure 5The two waveforms in the two recesses are the two actions of the patient touching the nose, and the embodiment of the application only needs to collect the action from the hand being lifted to the hand touching the nose, and other irrelevant information does not need to be considered. By finding the minimum value of each waveform, the start of the waveform is found from the minimum value one by one, as shown in the figure, the two red lines are the start and end of the unilateral hand touching the nose action, and the part of the sequence in the sequence of the bone key point data obtained from the collected video can be extracted to obtain the corresponding effective video segment of the left upper limb predetermined motion process and the corresponding effective video segment of only completing the right upper limb predetermined motion process, that is, step A11 is completed. Figure 5

[0110] Then step A12 is performed, and the predetermined points of each bone key point of the upper limb of the stroke patient include: left hand, left elbow, left shoulder, right hand, right elbow, right shoulder, that is, there are 6 bone key points.

[0111] For the effective video segment corresponding to the left upper limb predetermined motion process, the 6 bone key point data in each frame are arranged in order, such as the order [left hand, left elbow, left shoulder, right shoulder, right elbow, right hand], and 5 vectors are obtained according to the bone key point data of the front and back adjacent bones after sorting, that is, a vector is obtained by the left hand and the left elbow, a vector is obtained by the left elbow and the left shoulder, a vector is obtained by the left shoulder and the right shoulder, a vector is obtained by the right shoulder and the right elbow, and a vector is obtained by the right elbow and the right hand; the included angle between each two vectors (including the included angle between each vector and itself) is calculated, so 5 vectors can obtain 25 included angles, the 25 included angles obtained by the frame are sorted, the order can be in any predetermined order, which is not limited here, to obtain the included angle sequence corresponding to the frame, and the included angle sequences obtained by all frames are connected in the order of the frames to obtain the effective motion data sequence of the left side.

[0112] For the effective video segment corresponding to the right upper limb predetermined motion process, it is similar to the left side, and it needs to be noted that the bone key point data sorting method, the included angle sorting method, and the included angle sequence connection method are consistent with the left side.

[0113] Step A2, using flip flip technology to mirror the effective motion data sequence of one side of the upper limb to obtain the effective motion data sequence of the opposite side;

[0114] Among them, the mirror image of the effective motion data sequence of the left side can also be the effective motion data sequence of the right side.

[0115] Step A3, comparing the effective motion data sequence of the other side of the upper limb which is not mirror imaged and the opposite side effective motion data sequence by using the DTW algorithm to obtain the difference degree measurement value;

[0116] ​In assessing the progress of a patient's rehabilitation, comparing the similarity of movement patterns over different time periods is an important task. To achieve this, embodiments of the invention employ the Dynamic Time Warping (DTW) algorithm to compute the similarity (or dissimilarity) between movement trajectories. The DTW algorithm can compare the similarity between sequences of different lengths by aligning the time axes of the two sequences through non-linear time warping, thus providing a more accurate measure of their similarity.

[0117] In this process, the DTW algorithm considers the position and time of each point in the effective movement data sequences on both sides to find the optimal matching way. Through the DTW algorithm, a distance or similarity value can be obtained, which reflects the degree of similarity between the left and right hand nose touching movements.

[0118] The processing process of the DTW algorithm can be seen in Figure 6 The DTW algorithm includes the following steps:

[0119] Step 1: Initialize a two-dimensional matrix with a size of the product of the lengths of the two sequences, and initialize all elements to infinity. This matrix is used to store the cumulative distance between the two sequences.

[0120] Step 2: Initialize the first row and first column of the matrix to 0. These values represent the alignment of one sequence with the other when one sequence is empty, and the cumulative distance is 0.

[0121] Step 3: Starting from the top left corner of the matrix, calculate the distance between the corresponding points in the two sequences and add it to the cumulative distance at the corresponding position. Then, update the minimum cumulative distance values of the three adjacent positions around this position, which are the position above, the position to the left, and the position above and to the left.

[0122] Step 4: Repeat step 3 until the entire matrix is traversed. Finally, the right bottom corner element of the matrix is the minimum cumulative distance between the two sequences, i.e. the DTW distance between them, which is recorded as the dissimilarity measure value here.

[0123] By calculating the DTW distance of the upper limb movement trajectories of the patient in different time periods, the rehabilitation progress of the patient can be evaluated. If the DTW distance gradually decreases, it means that the patient's movement pattern is getting closer to the normal state, and the rehabilitation effect is good. On the contrary, if the DTW distance does not change significantly or even increases, it means that the patient's rehabilitation progress is slow or stagnant, and the rehabilitation treatment plan needs to be adjusted in time. By using the DTW algorithm to compare and analyze the movement data of multiple patients, the similarity of their movement patterns can be evaluated by calculating the DTW distance of the movement trajectories between different patients, so as to find potential movement disorders or abnormalities. This helps doctors make more accurate diagnoses and rehabilitation plans, and improves the rehabilitation effect and quality of life of patients.

[0124] The embodiment of the present application calculates the similarity between movement trajectories by using the DTW algorithm, which provides strong support for evaluating the rehabilitation progress of patients and developing personalized rehabilitation treatment plans. At the same time, this method can also be used to compare and analyze the movement data of multiple patients, providing useful reference information for clinical diagnosis and treatment.

[0125] Step A4, determining the NIHSS score according to the difference degree measurement value to obtain the upper limb movement evaluation result of the stroke patient.

[0126] After obtaining the difference degree measurement value, the NIHSS score can be determined according to the preset mode, and the upper limb movement evaluation result of the stroke patient can be obtained by using the NIHSS score. The NIHSS score can be directly used as the upper limb movement evaluation result, or some analysis information can be added to the NIHSS score to form the upper limb movement evaluation result, such as adding some explanations of the score results, rehabilitation suggestions, etc., which are all reasonable.

[0127] According to the obtained difference degree measurement value, researchers can evaluate the difference degree of the left and right hand nose touching action. A smaller DTW distance means a higher similarity between the two sequences and a smaller difference degree, which indicates that the patient's upper limb function is less impaired. On the contrary, it may mean that the patient's upper limb function is severely impaired.

[0128] One optional implementation of the preset mode can be a pre-determined mapping relationship between the difference degree measurement value and the NIHSS score, which can be a table. The mapping relationship can be determined by a large number of difference degree measurement values and corresponding NIHSS scores, such as 0≤difference degree measurement value<250, NIHSS score is 0, 250≤difference degree measurement value<500, NIHSS score is 1… and so on.

[0129] Of course, the NIHSS score determined according to the preset mode by using the difference degree measurement value can also be determined by using a function fitted by a large number of samples, or obtained by using a deep learning model trained by a large number of samples, which is reasonable.

[0130] Applying this evaluation result to the upper limb function score of the patient can provide valuable guidance information for medical personnel. By comparing the difference degree of the left and right hand nose touching actions, medical personnel can more accurately understand the upper limb function condition of the patient, so as to formulate a personalized rehabilitation plan. This evaluation method based on video image processing and DTW algorithm not only improves the accuracy of evaluation, but also provides strong support for the formulation and adjustment of rehabilitation treatment plan.

[0131] For ease of understanding, a process diagram from video acquisition to NIHSS score is given in the manner of MediaPipe library combined with DTW algorithm, please refer to Figure 7 . Among them, it is assumed that the collected video contains 50 frames, and each frame feature represents the skeletal key point data extracted from the frame. Please refer to Figure 8 , a difference degree measurement value (i.e. similarity) is given in the manner of MediaPipe library combined with DTW algorithm.

[0132] Please refer to Figure 9 , Figure 9 for an example of NIHSS score obtained by different stroke patients in the embodiment of the application. During the process of NIHSS score, the difference between left and right hands, the left and right arm and trunk angles, the left and right hand nose touching time can be specifically considered, and the NIHSS score is obtained by comprehensively considering the differences in these aspects. It can be understood that these information can be obtained by analyzing and processing the original video, which will not be described in detail here.

[0133] 2) In another optional implementation, S3 can include the following steps:

[0134] Step B1, input the time sequence corresponding to the skeletal key point data extracted from the video into the pre-trained LSTM model to obtain the model difference degree measurement of the upper limb movement of the stroke patient;

[0135] Step B2, determine the NIHSS score based on the model difference degree measurement to obtain the upper limb movement evaluation result;

[0136] The LSTM model is trained by using sample time sequences corresponding to the bone key point data extracted from sample videos of different individuals performing predetermined movement processes in sequence on both sides of the upper limbs, and corresponding labels; the labels of different sample time sequences represent different model difference measurement results of both sides of the upper limbs of the corresponding individual; the LSTM model comprises three LSTM layers and an LSTM output layer, the three LSTM layers have an increasing number of LSTM units followed by a decreasing number of LSTM units; and the number of units of the LSTM output layer is dynamically set.

[0137] Considering that the upper limb movement is a continuous process in which each action is related to the previous action, the embodiment of the present application can introduce a long short-term memory network (LSTM) to analyze the upper limb movement sequence of the patient. LSTM (Long Short-Term Memory) is a special RNN (Recurrent Neural Network) that can process sequence data with long-term dependencies. In the task of human pose prediction or action recognition, LSTM can capture the temporal dependencies between pose coordinates and make predictions based on these relationships. Therefore, it performs well in processing time series data.

[0138] The structure diagram of the LSTM network is shown in Figure 10 and the related parameters are understood in combination with the related concepts, which are not described in detail here. Some concepts involved are briefly introduced below.

[0139] ReLU activation function: ReLU (Rectified Linear Unit) is a commonly used activation function that can accelerate the training process of the model and alleviate the problem of gradient vanishing. In the model, ReLU is used as the activation function of LSTM and fully connected layers.

[0140] Softmax output layer: The Softmax activation function is used in the output layer. The Softmax function can convert the output of the model into a probability distribution, so that the sum of the prediction probabilities of each action category is 1. This is very useful for the action recognition task, because it allows the model to directly output the probability of each action category.

[0141] Sequential model: The sequential model is a simple model construction method provided by Keras, which allows different network layers to be stacked together through the.add() method. In the code, the sequential model makes the definition process of the model clear and concise.

[0142] The LSTM model of the embodiment of the present application is improved on the basis of the traditional LSTM network. The LSTM model includes three LSTM layers and an LSTM output layer. This multi-level structure can capture different levels of context information in the input sequence. Among the three LSTM layers, the first LSTM layer can capture low-level patterns, and the subsequent LSTM layers can build higher-level representations based on these low-level patterns. Each of the three LSTM layers has a different number of units, which are 64, 128, and 64, respectively. The three LSTM layers have an increasing number of LSTM units first and then a decreasing number of LSTM units. This "increasing first and then decreasing" design may help the model to capture more information in the early stage and then perform more detailed processing in the later stage.

[0143] The number of units of the LSTM output layer is dynamically set according to actions.shape[0], which means that the model can adapt to different numbers of action categories. This flexibility enables the model to be easily applied to new tasks with different action sets.

[0144] When training the LSTM model, the embodiment of the present application uses labeled upper limb movement data, including normal movement and abnormal movement cases. These upper limb movement data are sample time series corresponding to the bone key point data extracted from sample videos of a plurality of stroke patients completing a predetermined movement process on both sides of the upper limbs in sequence. It can be understood as sample time series corresponding to bone key point data extracted from complete videos collected, and the corresponding label can be a corresponding model difference degree measurement result, which can be determined in advance by manual labeling or the like.

[0145] Through supervised learning, the LSTM model can learn the feature representation of normal and abnormal movement patterns. During model training, a stochastic gradient descent (SGD) optimization algorithm can also be used, and appropriate learning rate and iteration number can be set to complete the training.

[0146] Once the LSTM model is trained, it can be applied to real-time monitoring of the upper limb movement of a patient. By extracting bone key point data in a video stream and inputting it into the trained LSTM model, the model can capture the temporal dependency between these bone key point data and learn the dynamic feature representation of the patient's upper limb movement. According to these dynamic feature representations, a model difference degree measurement result is finally obtained, thereby completing step B1. The model difference degree measurement result represents the difference between the two upper limbs in completing the predetermined movement process, which is used for further analysis and evaluation, such as judging whether the patient's movement is abnormal, evaluating the patient's rehabilitation progress, etc.

[0147] Regarding step B2, the NIHSS score is determined based on the model difference degree measurement, and an upper limb movement evaluation result is obtained; the method for determining the NIHSS score based on the model difference degree measurement can be determined in a manner similar to step A4, and details are not described herein.

[0148] The upper limb movement evaluation method for stroke patients provided by the embodiment of the application first collects videos of the two upper limbs of a stroke patient successively completing a predetermined movement process; wherein the goal of the two upper limbs completing the predetermined movement process is that the movement trajectories formed by the two upper limbs are axisymmetric along the vertical midline of the stroke patient; then the skeletal key point data in the video is extracted; wherein the skeletal key point data represents the position information of each skeletal key point of the upper limbs of the stroke patient at the corresponding moment; finally, based on the feature difference of the two upper limbs in the predetermined movement process in the skeletal key point data, the NIHSS score of the stroke patient is determined, and an upper limb movement evaluation result is obtained.

[0149] In an optional implementation, the MediaPipe library and the preset network based on the ResNet50 model can also be combined to achieve more accurate skeletal key point extraction.

[0150] Traditional upper limb movement evaluation usually relies on subjective observation and manual testing by medical personnel, and therefore has the problems of subjectivity and inaccuracy. However, the application can realize objective and automatic detection of upper limb movement by using computer vision technology, thereby improving the objectivity and standardization of evaluation. Traditional upper limb movement evaluation often requires a large amount of time and human resources, which may be a burden for medical institutions and doctors, especially in a busy clinical work environment. The application can realize real-time detection and analysis of upper limb movement, greatly improving the efficiency and time effect of evaluation, and helping to discover changes in patient movement in a timely manner and carry out targeted rehabilitation intervention. The rehabilitation process of stroke patients is usually a continuous process that requires long-term tracking and evaluation. However, the application can realize continuous monitoring and tracking of the upper limb movement of patients, can record the movement state and change trend of patients in real time, and provide timely data support for doctors, which helps to develop individualized rehabilitation plans and adjust treatment plans.

[0151] Further, the embodiment of the application can use the MediaPipe library or the preset network based on the ResNet50 model to extract skeletal key points, and can use the DTW algorithm or the LSTM model to determine the NIHSS score, thereby obtaining an upper limb movement evaluation result. The application realizes real-time monitoring, analysis and evaluation of the upper limb movement of patients, not only improves the accuracy and objectivity of evaluation, but also greatly simplifies the operation process and improves the evaluation efficiency. It can better serve the treatment and rehabilitation needs of stroke patients.

[0152] In a second aspect, corresponding to the method embodiments described above, the embodiments of the present application also provide an upper limb movement assessment system for stroke patients, as shown in Figure 11 The upper limb movement assessment system comprises a user terminal, a server terminal and a database terminal; wherein:

[0153] The user terminal is configured to collect videos of the stroke patient's bilateral upper limbs performing a predetermined movement process in sequence and extract skeletal key point data in the videos; wherein the goal of the bilateral upper limbs performing the predetermined movement process is that the movement trajectories formed by the bilateral upper limbs are axisymmetric along the vertical midline of the stroke patient; and the skeletal key point data represents position information of each skeletal key point of the stroke patient's upper limbs at a corresponding time point;

[0154] The server terminal is configured to receive the skeletal key point data sent by the user terminal, determine the NIHSS score of the stroke patient based on the feature differences of the bilateral upper limbs in the predetermined movement process in the skeletal key point data, obtain an upper limb movement assessment result, and send the upper limb movement assessment result to the user terminal and the database terminal;

[0155] The database terminal is configured to store the upper limb movement assessment result of each stroke patient.

[0156] Specifically, the user terminal is the starting point of the entire system. In an optional embodiment, the user terminal can be provided with a front-end page, which can be designed using a programming language. Patients can understand the content of the test and the required actions under the guidance of the front-end page. This step is crucial because it directly determines whether the patient can accurately and normatively complete the test action. The front-end page not only provides clear and accurate guidance, but also has a friendly and easy-to-use interface design, so that patients can easily get started.

[0157] As an example structure of the system, please refer to Figure 12 . Figure 12 In the MediaPipe library combined with the DTW algorithm as an example. Figure 13 is a flowchart of starting video collection until obtaining the upper limb movement assessment result.

[0158] Regarding the system implementation process of different schemes such as the preset network based on the ResNet50 model and the LSTM model, no further illustration and description will be given here. The embodiments of the present application can select one of the MediaPipe library and the preset network based on the ResNet50 model as the way to extract the skeletal key points, and select one of the DTW algorithm and the LSTM model as the way to obtain the upper limb movement assessment result, and the specific embodiments are not limited here.

[0159] The patient records a video according to the guidance on the front page. During the recording process, the system uses the MediaPipe library to detect the skeletal key points in the video. These skeletal key points can reflect the patient's motion trajectory, posture changes, etc., which are crucial for subsequent analysis and evaluation. The detected skeletal key points are sent to the server side for further processing as video data.

[0160] On the server side, the system uses the dynamic time warping algorithm to analyze the video data and calculate the difference measure value between the left and right hands. This step is a key link in evaluating the patient's motor function, as it can objectively reflect the patient's motor coordination and balance ability. According to the size of the difference measure value, the system calculates the NIHSS (National Institutes of Health Stroke Scale) score, which is a scale used to evaluate the degree of neurological impairment in stroke patients. NIHSS can be used as the upper limb motor evaluation result of the stroke patient.

[0161] The calculated NIHSS score is stored in the database side for subsequent data analysis and processing. The design of the database side needs to consider data storage, query, and security, etc., to ensure the integrity and accuracy of the data. The NIHSS score in the embodiment of the present application can judge the patient's motor function status, including muscle strength, coordination, flexibility, etc., providing a reference and basis for medical personnel.

[0162] Optionally, the upper limb motor evaluation system is provided with a historical score acquisition interface, and the user side acquires the multiple NIHSS scores of the stroke patient from the database side through the service side by using the historical score acquisition interface, so as to analyze the NIHSS score trend.

[0163] Specifically, the system provides a historical score acquisition interface, and the user can acquire the multiple NIHSS score data of the patient through the server side. These data can be used to evaluate the rehabilitation effect of the patient and provide strong support for formulating personalized rehabilitation plans for the user. At the same time, through statistical analysis of a large amount of data, doctors can also find the rules and trends in the patient's rehabilitation process, providing a scientific basis for further optimizing the treatment scheme.

[0164] The score results are saved to the database end, and a historical score curve of the patient can be formed. Examples of the historical score curve can be seen in FIGS. 14(a) and 14(b). The historical curve part visually displays the evaluation results of the patient in the form of a chart or graph. This can include the trend of the upper limb movement score of the patient over time, the change of different evaluation indicators, etc., enabling medical personnel to intuitively understand the rehabilitation progress of the patient. The historical curve part can also perform trend analysis on the rehabilitation progress of the patient. Medical personnel can understand the speed and direction of the patient's rehabilitation and the effectiveness of the treatment plan by observing the change trend of the historical curve, and provide more personalized and effective rehabilitation treatment recommendations for the patient. The historical curve part allows medical personnel to view and review the evaluation history of the patient at any time. This helps medical personnel to comprehensively understand the changes in the patient's condition and the rehabilitation progress, and provides a scientific basis for the development of diagnosis, treatment and rehabilitation plans.

[0165] Through the database end, the score data of the patient can be conveniently stored, queried and analyzed, providing medical personnel with more comprehensive and detailed patient information, which helps to develop personalized rehabilitation treatment plans and track rehabilitation progress.

[0166] For specific processing procedures of each part of the system, please refer to the related content of the first aspect, which will not be repeated here.

[0167] The upper limb movement evaluation system for stroke patients provided by the embodiments of the present application uses computer vision technology to provide upper limb movement evaluation for stroke patients, which can provide medical personnel with more comprehensive and objective patient information, and helps to develop personalized rehabilitation treatment plans and track rehabilitation progress. The present application adopts a variety of technical solutions, combines computer vision, deep learning, database and other technologies, and realizes real-time monitoring and analysis of the upper limb movement of stroke patients. This not only improves the efficiency and time effect of evaluation, but also helps to discover the movement changes of patients in time and carry out targeted rehabilitation intervention. At the same time, the present application also has good scalability and customizability, and can be improved and optimized according to specific needs. It is believed that this technology will have a positive impact on the rehabilitation treatment and health management of stroke patients.

[0168] Each embodiment in the specification is described in a related manner, and the same and similar parts between each embodiment can be referred to each other. Each embodiment mainly explains the difference from other embodiments. Especially for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0169] The above merely provides the preferred embodiments of the application, and not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A method for evaluating upper limb movement of a stroke patient, characterized by, The method comprises: collecting videos of a stroke patient performing a predetermined movement process with both upper limbs in sequence; wherein the goal of the two upper limbs performing the predetermined movement process is that the movement trajectories formed by the two sides are axisymmetric along the vertical midline of the stroke patient; extracting bone key point data in the video; wherein the bone key point data represents the position information of each bone key point of the upper limbs of the stroke patient at the corresponding time; based on the feature difference of the two upper limbs in the predetermined movement process in the bone key point data, determining the NIHSS score of the stroke patient to obtain the upper limb movement evaluation result, comprising: using the bone key point data to determine the corresponding effective movement data sequence of each side of the stroke patient performing the predetermined movement process; using flip flip technology to mirror the effective movement data sequence of one side of the upper limb to obtain the contralateral effective movement data sequence; using the DTW algorithm to compare the sequence similarity of the effective movement data sequence of the other side of the upper limb which is not mirror and the contralateral effective movement data sequence, to obtain the difference degree measurement value; determining the NIHSS score according to the difference degree measurement value to obtain the upper limb movement evaluation result of the stroke patient.

2. The upper limb motor assessment method according to claim 1, characterized in that, The predetermined movement process of the unilateral upper limb comprises: the hand starts from the natural position falling on the side of the body, goes through a plurality of predetermined positions, and finally touches the nose tip.

3. The upper limb motor assessment method according to claim 1, wherein, The bone key point data in the video is extracted by using the MediaPipe library.

4. The upper limb motor assessment method of claim 1, wherein, The bone key point data in the video is extracted by using a preset network based on a ResNet50 model; wherein the preset network is obtained by training using a picture data set with node annotation information; The preset network comprises a preset convolutional layer, a feature extraction module, a two-dimensional global average pooling layer, a Dropout layer and a Dense layer connected in sequence; wherein the preset convolutional layer is used to adjust the dimension of the input data, the preset convolutional layer is a two-dimensional convolutional layer, uses a 1x1 Kernel kernel, and uses a LeakyReLU activation function; the feature extraction module is composed of the convolutional layer in the ResNet50 model; the two-dimensional global average pooling layer is used to aggregate features; the Dropout layer and the Dense layer are fully connected layers, and the Dropout layer is used to reduce overfitting; the Dense layer is used as the output layer of the preset network.

5. The upper limb motor assessment method according to claim 1, wherein, Each bone key point of the upper limbs of the stroke patient comprises: a left hand, a left elbow, a left shoulder, a right hand, a right elbow, and a right shoulder.

6. The upper limb motor assessment method of claim 1, wherein, Using the bone key point data, the corresponding effective movement data sequence of each side of the stroke patient performing the predetermined movement process is determined, comprising: based on the bone key point data determined in each frame of the video, determining the corresponding effective video segment when the stroke patient only completes the predetermined movement process of the left upper limb and the corresponding effective video segment when the stroke patient only completes the predetermined movement process of the right upper limb; For each side of the effective video segment, arrange all the skeletal key point data in each frame in sequence, obtain a plurality of vectors according to the skeletal key point data before and after the sequence, calculate the included angle between each two vectors to obtain the corresponding included angle sequence of the frame, connect the included angle sequences obtained from all the frames in the order of the frames to obtain the effective motion data sequence of the corresponding side.

7. The upper limb motor assessment method of claim 1, wherein, Based on the feature difference of the upper limbs of the two sides in the predetermined motion process in the skeletal key point data, the NIHSS score of the stroke patient is determined, and an upper limb motion evaluation result is obtained, including: Inputting the time sequence corresponding to the skeletal key point data extracted from the video into a pre-trained LSTM model to obtain a model difference degree measurement of the upper limb motion of the stroke patient; Based on the model difference degree measurement, the NIHSS score is determined to obtain the upper limb motion evaluation result; Wherein, the LSTM model is trained by using the sample time sequence corresponding to the skeletal key point data extracted from the sample video of the two sides of the upper limbs of different individuals completing the predetermined motion process in sequence, and the corresponding label; the label of different sample time sequences represents the different model difference degree measurement results of the two sides of the upper limbs of the corresponding individual; the LSTM model includes three LSTM layers and an LSTM output layer, and the three LSTM layers have an increasing number of LSTM units first and then decreasing; the number of units of the LSTM output layer is dynamically set.

8. A system for assessing upper extremity movement of a stroke patient, comprising: It includes a user end, a server end and a database end; wherein: The user end is used to collect the video of the stroke patient when the two sides of the upper limbs complete the predetermined motion process in sequence, and extract the skeletal key point data in the video; wherein, the goal of the two sides of the upper limbs completing the predetermined motion process is that the motion trajectories formed by the two sides are axisymmetric along the vertical midline of the stroke patient; the skeletal key point data represents the position information of each skeletal key point of the upper limbs of the stroke patient at the corresponding time; The server end is used to receive the skeletal key point data sent by the user end, determine the NIHSS score of the stroke patient based on the feature difference of the two sides of the upper limbs in the predetermined motion process in the skeletal key point data, obtain the upper limb motion evaluation result, and send the upper limb motion evaluation result to the user end and the database end; wherein, based on the feature difference of the two sides of the upper limbs in the predetermined motion process in the skeletal key point data, the NIHSS score of the stroke patient is determined, and the upper limb motion evaluation result is obtained, including: using the skeletal key point data, determining the effective motion data sequence corresponding to each side of the upper limbs of the stroke patient when performing the predetermined motion process; using flip flip technology to mirror flip the effective motion data sequence of one side of the upper limbs to obtain the effective motion data sequence of the opposite side; using the DTW algorithm to compare the sequence similarity of the effective motion data sequence of the other side of the upper limbs which is not mirror flipped and the effective motion data sequence of the opposite side to obtain a difference degree measurement value; determining the NIHSS score according to the difference degree measurement value to obtain the upper limb motion evaluation result of the stroke patient; The database end is used for storing the upper limb movement assessment results of each stroke patient.

9. The upper limb motor assessment system of claim 8, wherein, The upper limb movement assessment system is provided with a historical score acquisition interface, and the user end uses the historical score acquisition interface to acquire the multiple NIHSS scores of the stroke patient from the database end through the server end, so as to analyze the NIHSS score trend.

Images