Hand Function Rehabilitation Evaluation System and Its Application Method

Abstract

This application discloses a hand function rehabilitation evaluation system and method. The system includes a hand evaluator and a main controller. The hand evaluator collects a first video signal of a patient's palm completing a first specified test process. The main controller obtains a first characteristic parameter according to the first video signal, compares it with a corresponding first target parameter value, and obtains a hand function rehabilitation evaluation result. Among them, the first specified test includes the palm moving from a closed state to an open state, or the palm moving from an open state to a closed state. The first characteristic parameter includes the extended length of each finger, and the time interval between when the extended lengths of all fingers reach the maximum reference value and when the extended lengths of all fingers reach the minimum reference value. In the embodiments of this application, the hand evaluator can collect the video signal of the patient's palm completing the specified test process in real time, process these signals in a timely manner and extract characteristic parameters, so as to quickly obtain the hand function rehabilitation evaluation result. Through precise comparison, the evaluation result is more objective and accurate.

Description

Technical Field

[0001] This application belongs to the technical field of robot system control, and specifically relates to a hand function rehabilitation evaluation system and its application method. Background Art

[0002] In the research on human limb movement, the clinical evaluation methods and the industrial evaluation methods have different tendencies in methodology.

[0003] The limb rehabilitation evaluation scales designed by clinical researchers usually design several groups of movement behaviors. While the patient completes the corresponding movements, the physician uses intuitive methods such as observation, inquiry, and manual feeling to confirm the patient's rehabilitation rating. These methods have been tried and corrected by predecessors for many years and have become the gold standard in related fields. However, these traditional hand function rehabilitation evaluation methods are easily affected by the personal experience and skill level of the evaluator, resulting in inconsistencies and subjectivity in the evaluation results, and often taking a long time with a relatively cumbersome operation process.

[0004] In contrast, the industrial evaluation methods usually indirectly observe the signal changes generated by the patient's limb movement based on the sensor data of the designed observation system. These methods have certain innovation and the recording process is fast and convenient. However, due to interdisciplinary and latecomer advantages, the industrial evaluation methods still need to be further improved, and there are also diversities in the methods of feature extraction and evaluation result classification. Therefore, it is still a challenge to obtain the best results in terms of accuracy and so on. Summary of the Invention

[0005] The technical purpose of this application is to propose a hand function rehabilitation evaluation system and its application method to take into account the advantages of traditional clinical evaluation methods and the efficiency of new industrial evaluation methods, and improve the accuracy of hand rehabilitation evaluation.

[0006] To achieve the above technical purpose, this application adopts the following technical solutions.

[0007] In the first aspect, an embodiment of this application provides a hand function rehabilitation evaluation system, including a hand evaluator and a main controller, and the hand evaluator and the main controller are communicatively connected;

[0008] The hand evaluator is configured to collect a first video signal of a patient's palm completing a first specified test process;

[0009] The main controller is configured to extract first image data covering a preselected area of the palm according to the first video signal, mark the positions of the palm joint points based on the first image data, confirm a first characteristic parameter of the palm completing the first specified test process based on the marking result, compare the first characteristic parameter with a corresponding first target parameter value, and obtain a hand function rehabilitation evaluation result;

[0010] Among them, the first specified test includes the palm moving from closing to opening, or the palm moving from opening to closing; the first characteristic parameter includes the extending length of each finger, and the time interval between when the extending lengths of all fingers reach the maximum reference value and when the extending lengths of all fingers reach the minimum reference value.

[0011] In some embodiments, the hand evaluator includes a first imaging device and a collection device body. The first imaging device is disposed in the collection device body, and the collection device body is provided with a collection window corresponding to the position of the first imaging device, so that the first imaging device can collect the first video signal through the collection window.

[0012] In some embodiments, the method for determining the extending length of each finger is as follows:

[0013] According to the annotation result, obtain the vertical and horizontal coordinates of the fingertip joints of each finger at the current collection moment, as well as the vertical and horizontal coordinates of the fingertip joints of each finger at the previous collection moment, and determine the change amounts of the vertical and horizontal coordinates of the fingertip joints of each finger.

[0014] Determine the extending length of each finger according to the change amount, and the expression is:

[0015]

[0016]

[0017] Among them, X0 len represents the extending length of the thumb at the current collection moment, X1 len represents the extending length of the index finger at the current collection moment, X2 len represents the extending length of the middle finger at the current collection moment, X3 len represents the extending length of the ring finger at the current collection moment, X4 len represents the extending length of the little finger at the current collection moment, x00 and y00 are respectively the horizontal and vertical coordinates of the palm root, x0 n , y0 n are respectively the horizontal and vertical coordinates of the thumb fingertip at the current collection moment, x1 n , y1 n are respectively the horizontal and vertical coordinates of the index finger fingertip at the current collection moment, x2 n , y2 n are respectively the horizontal and vertical coordinates of the middle finger fingertip at the current collection moment, x3 n , y3 n are respectively the horizontal and vertical coordinates of the ring finger fingertip at the current collection moment, x4 n , y4 nrepresent the abscissa and ordinate of the little finger fingertip at the current acquisition moment; x0 r and y0 r respectively represent the abscissa change amount and ordinate change amount of the thumb fingertip, x1 r and y1 r respectively represent the abscissa change amount and ordinate change amount of the index finger fingertip at the current acquisition moment, x2 r and y2 r respectively represent the abscissa change amount and ordinate change amount of the middle finger fingertip at the current acquisition moment, x3 r and y3 r respectively represent the abscissa change amount and ordinate change amount of the ring finger fingertip at the current acquisition moment, x4 r and y4 r respectively represent the abscissa change amount and ordinate change amount of the little finger fingertip at the current acquisition moment.

[0018] In some embodiments, determining the time interval between when the extended lengths of all fingers reach the maximum reference value and when the extended lengths of all fingers reach the minimum reference value includes:

[0019] Presetting the maximum reference value and minimum reference value of the extended length of each finger;

[0020] Detecting the extended length of each finger in real time at a preset sampling rate;

[0021] If the extended length of any finger exceeds the maximum reference value or the minimum reference value, update the maximum reference value and the minimum reference value;

[0022] Record the first moment when the extended lengths of all fingers reach the maximum reference value, and the second moment when the extended lengths of all fingers reach the minimum reference value, and determine the difference between the first moment and the second moment as the time interval.

[0023] In some embodiments, the system further includes a finger-to-nose test evaluator communicatively connected to the main controller,

[0024] The finger-to-nose test evaluator is configured to collect a second video signal of the patient's palm completing the second specified test process;

[0025] The main controller is configured to extract second feature parameters according to the second video signal; compare the second feature parameters with corresponding second target parameter values to obtain a hand function rehabilitation evaluation result;

[0026] Wherein, the second specified test includes a finger-to-nose test, and the second feature parameters include the time taken for the second specified test process and the state after the palm reaches the nose position.

[0027] In some embodiments, the finger-to-nose tester includes a mobile platform, a telescopic rod, and a second imaging device. The telescopic rod is disposed on the mobile platform and can be telescoped in a direction perpendicular to the ground; the second imaging device is mounted on the telescopic rod.

[0028] In some embodiments, extracting a second feature parameter according to the second video signal includes:

[0029] Based on each frame image of the second video signal, extracting second image data covering a preselected area of the palm; based on the second image data, annotating the positions of the palm joint points and the nose tip joint points;

[0030] Based on the annotation result, determining the distance between the palm joint point and the nose tip joint point; when the distance is less than a set distance threshold, determining that the palm reaches the nose tip position; determining the time used for the second specified test process as the difference between the time when the palm last reached the nose tip position and the time when the palm reaches the nose tip position this time; judging the state of the palm based on the coordinate change of the palm joint point after the palm reaches the nose position, where the state includes whether there is tremor.

[0031] In some embodiments, the system further includes a grip strength evaluator communicatively connected to the main controller, and the grip strength evaluator is configured to collect the grip strength value of the grip strength evaluator.

[0032] In a second aspect, based on the hand function rehabilitation evaluation system provided in the above embodiments, the present application embodiment further includes:

[0033] A hand activity instrument, configured to include a mechanical finger structure and be wearable, drive a motor according to an output behavior value instruction sent by the main controller to drive the mechanical finger structure to perform corresponding bending actions, and feedback the grip strength value of the hand activity instrument during the bending actions of the mechanical finger;

[0034] A grip strength evaluator communicatively connected to the main controller;

[0035] Wherein, the hand evaluator is configured to collect a first video signal of the healthy side palm of the patient completing a first specified test process;

[0036] The grip strength evaluator is configured to collect the grip strength value of the grip strength evaluator after the patient wears the hand activity instrument.

[0037] In a third aspect, an application method of the hand function rehabilitation evaluation system provided in any feasible implementation manner of the second aspect of the present application includes:

[0038] Placing the healthy side hand of the patient on the hand evaluator and collecting a first video signal of the patient's palm completing a first specified test process;

[0039] The main controller is used to extract first image data covering the preselected area of the palm according to the first video signal, mark the positions of the palm joint points based on the first image data, confirm the extended lengths of the fingers based on the marking results, and update the reward and punishment value of the hand evaluator according to the extended lengths.

[0040] The grip strength evaluator is used to obtain the grip strength value of the grip strength evaluator in real time; calculate the reward and punishment value of the grip strength evaluator according to the grip strength value of the grip strength evaluator; and calculate the value score of the grip strength evaluator. If the value score of the grip strength evaluator is less than or equal to the value score threshold, update the reward and punishment value of the grip strength evaluator.

[0041] Let the affected palm of the patient wear the hand activity instrument, place the grip strength evaluator inside the mechanical finger structure, and the mechanical finger structure does not contact the grip strength evaluator.

[0042] The hand activity instrument obtains the output behavior value instruction of the previous round, drives the mechanical finger structure to hold the grip strength evaluator according to the output behavior value instruction, and the main controller reads the grip strength value of the grip strength evaluator fed back by the grip strength evaluator and the grip strength value of the hand activity instrument fed back by the hand activity instrument; detect whether the grip strength value of the grip strength evaluator reaches the standard.

[0043] If it does not reach the standard, update the output behavior value instruction to drive the hand activity instrument according to the collected grip strength value of the grip strength evaluator, the grip strength value of the hand activity instrument, the target grip strength value of the grip strength evaluator, the target grip strength value of the hand activity instrument, and the current reward and punishment value of the hand evaluator and the reward and punishment value of the grip strength evaluator.

[0044] If it reaches the standard, the main controller drives the hand activity instrument to release the grip strength evaluator.

[0045] Compared with the prior art, the hand function rehabilitation evaluation system provided by the embodiment of the present application can collect the video signal of the patient's palm completing the specified test process in real time through the hand evaluator, and the main controller can process these signals in real time and extract the characteristic parameters, so as to quickly obtain the hand function rehabilitation evaluation result. This real-time performance helps to timely understand the rehabilitation situation of the patient and provide timely feedback for rehabilitation treatment; by accurately comparing the characteristic parameters extracted from the video signal with the preset target parameter values, the evaluation result is more objective and accurate. This method reduces the influence of subjective factors on the evaluation result and improves the credibility and consistency of the evaluation.

[0046] Compared with the prior art, the application method of the hand function rehabilitation evaluation system provided by the embodiments of the present application has more objective and accurate evaluation results. This method reduces the influence of subjective factors on the evaluation results, improves the credibility and consistency of the evaluation, and is conducive to the patient to drive the hand activity instrument through the healthy palm for movement, so that the grip force evaluation of each pinching action reaches the required level, and then gradually makes the affected palm approach the required grip force level. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure of the present application in any way. In addition, the shapes and proportional dimensions of the components in the drawings are only schematic and are used to assist in understanding the present application, rather than specifically limiting the shapes and proportional dimensions of the components of the present application. Those skilled in the art can, under the teaching of the present application, select various possible shapes and proportional dimensions according to specific circumstances to implement the present application. In the drawings:

[0048] Figure 1 is a structural block diagram of a hand evaluator in the hand function rehabilitation evaluation system according to an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of the extended length of the palm according to an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of the update process of the maximum reference value and the minimum reference value according to an embodiment of the present application;

[0051] Figure 4 is a structural block diagram of a finger-to-nose test evaluator in the hand function rehabilitation evaluation system according to an embodiment of the present application;

[0052] Figure 5 is a schematic diagram of a human pose estimation model used in another embodiment of the present application;

[0053] Figure 6 is a structural block diagram of a grip strength tester in the hand function rehabilitation evaluation system according to an embodiment of the present application;

[0054] Figure 7 is a side view block diagram of the structure of the hand function rehabilitation evaluation system in another embodiment of the present application;

[0055] Figure 8 is a front view block diagram of the structure of the hand function rehabilitation evaluation system in another embodiment of the present application;

[0056] Figure 9 is a schematic diagram of the application method flow of the hand function rehabilitation evaluation system provided by the embodiment of the present application;

[0057] Reference Signs:

[0058] 1 - Hand Evaluator, 2 - Finger - to - Nose Test Evaluator, 3 - Grip Strength Tester, 4 - Display, 5 - Palm, 6 - Patient, 7 - Hand Activity Instrument, 11 - First Camera Device, 12 - Main Body of the Acquisition Device, 13 - Acquisition Window, 21 - Second Camera Device, 22 - Telescopic Rod, 23 - Support Rod, 24 - Mobile Platform, 25 - Universal Wheel, 61 - Affected - side Palm, 62 - Unaffected - side Palm. Detailed Implementation Manner

[0059] In order to enable those skilled in the art of this technology to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0060] In the description of this application, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more features.

[0061] Currently, there is a set of guiding principles for the medical standards of hand rehabilitation assessment. These guiding principles are derived from practical experience and scientific research applied in clinical practice, are widely accepted, and are considered the best practices in the industry. In hand rehabilitation assessment, medical standards usually include assessment scales and scoring systems for various limb movement behaviors. These scales and scoring systems have been clinically verified and revised over the years, are adopted in extensive clinical practice, and have become the gold standard for evaluating the effect of hand rehabilitation. The mainstream medical assessment scale benchmarks can generally be classified according to muscle changes, movement patterns, and hand function changes.

[0062] However, in these traditional hand - function rehabilitation assessment methods, they often rely on the experience and subjective judgment of doctors or therapists, are easily affected by the personal experience and skill level of the assessors, resulting in inconsistency and subjectivity of the assessment results, and often take a long time and the operation process is relatively cumbersome.

[0063] The hand function rehabilitation evaluation system provided by the embodiments of the present application can collect the video signals of the patient's palm during the completion of the specified test process in real time through the hand evaluator. The main controller can process these signals in a timely manner and extract characteristic parameters, so as to quickly obtain the hand function rehabilitation evaluation result. This real-time nature helps to understand the patient's rehabilitation situation in a timely manner and provide timely feedback for rehabilitation treatment; by accurately comparing the characteristic parameters extracted from the video signals with the preset target parameter values, the evaluation result is more objective and accurate. This method reduces the influence of subjective factors on the evaluation result and improves the credibility and consistency of the evaluation.

[0064] The following will describe in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

[0065] The embodiments of the present application provide a hand function rehabilitation evaluation system, including a hand evaluator and a main controller, and the hand evaluator and the main controller are communicatively connected; the hand evaluator is configured to collect the first video signal of the patient's palm during the completion of the first specified test process; the main controller is configured to extract the first image data covering the preselected area of the palm according to the first video signal, mark the positions of the palm joint points based on the first image data, confirm the first characteristic parameters of the palm during the completion of the first specified test process based on the marking result, compare the first characteristic parameters with the corresponding first target parameter values, and obtain the hand function rehabilitation evaluation result; wherein, the first specified test is that the palm moves from closing to opening, or the palm moves from opening to closing; the first characteristic parameters include the extended lengths of each finger, and the time interval between when the extended lengths of each finger reach the maximum reference value and when the extended lengths of each finger reach the minimum reference value.

[0066] Communication connection means that through the transmission and interaction of signals, communication is formed between the connected devices to complete the transmission and exchange of information. The communication connection can include two forms: wired connection and wireless connection. The wired connection is usually realized through physical media such as cables and optical fibers, while the wireless connection uses wireless signals such as electromagnetic waves for transmission.

[0067] In some embodiments, the structural block diagram of the hand evaluator 1 is as Figure 1 shown. The hand evaluator 1 includes a first imaging device 11 and a collection device body 12. The first imaging device 11 is placed in the collection device body 12, and the collection device body 12 is provided with a collection window 13 corresponding to the position of the first imaging device 11, so that the first imaging device 11 can collect the first video signal through the collection window 13.

[0068] In some embodiments, the acquisition device body 12 is an airtight PVC (Polyvinyl chloride) box. A transparent mounting cover may be provided on the top of the acquisition device body 12 to form a transparent acquisition window 23. The height of the PVC box is D1. During use, a use distance D2 is maintained between the palm 5 and the acquisition window 23 formed by the transparent mounting cover. This distance ensures that the first imaging device 11 can obtain a sufficient focusing distance to provide correct imaging for the main controller ( Figure 1 not shown in the figure), and the patient can obtain different evaluation levels by opening and closing the palm 5.

[0069] In a specific embodiment, the main controller may be a device with data processing and analysis capabilities, such as a personal computer (PC), an industrial control computer, an embedded system, a server, a tablet computer, or a notebook, etc.

[0070] In some embodiments, the video stream recorded by the first imaging device 11 passes through the hand detector in the algorithm program of the main controller and will be marked as a rough region of interest covering the entire palm 5, and then transmitted backward to the detection annotator. An accurate hand rectangular box will be generated in the detection annotator, and the accurate range of the hand is marked in this hand rectangular box.

[0071] The frame image marked with the hand rectangular box is further cropped to obtain a rough hand model analysis frame. This frame will be sent to the joint annotator for palm joint coordinate annotation. If the frame is too large, it will be marked with the region of interest and then cropped again. If the data recognition of the joint annotator is successful, annotation will be performed to generate a palm joint model. In the palm joint model established with the palm root as the origin of the polar coordinate system of the palm joints, the screen coordinates of all finger joints of the palm can be obtained.

[0072] In the example as Figure 2 shown, actually only the coordinates of the palm root and the coordinates of each fingertip are needed to determine the extended length of the palm. R1, R2, and R3 are the extended lengths of the fingers in the palm joint model in this coordinate system. If the fist is clenched, the values of R1, R2, and R3 will correspondingly decrease. If the palm is extended, the values of R1, R2, and R3 will correspondingly increase.

[0073] The following is the basic composition of the recognition result matrix of the extended length of the fingers. This matrix is two-dimensional, with 5 rows and 2 columns. The basic row composition is 1 row and 2 columns. The first dimension stores the abscissa x of the fingertip (tip) joint of each finger in the original frame image, and the second dimension stores the ordinate y of the fingertip joint of each finger in the original frame image.

[0074] For example, x0 n represents the abscissa of the latest thumb fingertip joint in the current refresh process, and x1 nIndicates the abscissa of the latest index finger fingertip joint in the current refresh process, and encodes each finger in sequence thereafter; x00 and y00 are the abscissa and ordinate of the palm root. The result matrix R is the distance between the fingertip joints of each finger and two points in the current polar coordinate system. Then x0 r , y0 r are respectively the coordinate change degrees of the thumb fingertip joint. It is easy to know that the change radius R of the thumb is The following formulas (1) and (2) show the method for calculating the extension length of each finger.

[0075]

[0076] In some embodiments, the first characteristic parameter further includes the time interval between when the extension lengths of each finger reach the maximum reference value and when the extension lengths of each finger reach the minimum reference value. That is, when the distal joint (fingertip joint) first reaches the maximum reference value, that is, when the extension lengths of the fingers are all the largest, start timing, and stop timing until the fingertip reaches the proximal joint (i.e., the palm root), that is, when the extension lengths of the fingers all reach the minimum reference value. Determine the time interval between the two timings, and compare the time interval confirmed by timing with the preset time interval to obtain the hand function rehabilitation evaluation result.

[0077] In some embodiments, as Figure 3 shown, determining the time interval between when the finger joint reaches the maximum reference value and when it reaches the minimum reference value includes: presetting the maximum reference value and the minimum reference value of the extension length of each finger in the reference coordinate system; detecting the extension length of each finger in real time at the preset sampling rate; if there is a finger whose extension length exceeds the maximum reference value or the minimum reference value, update the maximum reference value and the minimum reference value; the extension length of the finger that exceeds the maximum reference value or the minimum reference value can be used as the updated maximum reference value or minimum reference value; record the first moment when the extension lengths of each finger all reach the maximum reference value, and the second moment when the extension lengths of each finger all reach the minimum reference value, and determine the difference between the first moment and the second moment as the time interval.

[0078] The embodiments of the present application update the maximum reference value and the minimum reference value in real time, can gradually adapt to and learn the finger movements of the user, can adapt to different rehabilitation stages of the patient, and improve the user experience and personalized service.

[0079] Some embodiments provide a hand function rehabilitation evaluation system, further comprising a finger-to-nose test evaluator communicatively connected to the main controller, the finger-to-nose test evaluator being configured to collect a second video signal of a patient's palm completing a second specified test process; a main controller configured to extract a second feature parameter according to the second video signal; compare the second feature parameter with a corresponding second target parameter value to obtain a hand function rehabilitation evaluation result; wherein the second specified test includes a finger-to-nose test, and the second feature parameter includes the time taken for the second specified test process and the state after the palm reaches the nose position.

[0080] In the study of human limb movement, the finger-to-nose test is a commonly used clinical evaluation method. In clinical evaluation, the finger-to-nose test includes the patient being required to straighten the forearm and touch their own nose tip with the index finger, first slowly and then quickly, first with eyes open and then with eyes closed, and repeat this action. Under normal circumstances, the finger-to-nose movement should be accurate and smooth. However, if there are problems with motor coordination or muscle control, the patient's finger-to-nose movement may appear clumsy, inaccurate, uncoordinated or unsteady. But this clinical evaluation method often takes a long time, has a high usage cost, and the operation process is relatively cumbersome.

[0081] As Figure 4 shown, in the hand function rehabilitation evaluation system provided by the embodiments of the present application, the finger-to-nose test evaluator 2 includes a mobile platform 24, a telescopic rod 22 and a second imaging device 21. Optionally, a support rod 23 is provided on the mobile platform 14, the support rod 23 is provided with a telescopic rod 22, and the telescopic rod 22 can be telescoped along a direction perpendicular to the ground; the second imaging device 21 is installed on the telescopic rod 22. Universal wheels 25 are installed at the bottom of the mobile platform 24.

[0082] In some embodiments, the main controller is disposed on the mobile platform 24.

[0083] As an example, the hand function rehabilitation evaluation system may further include a display 4 communicatively connected to the main controller, and the display 4 can be used to display data such as specified test prompts and test results.

[0084] In the embodiment, the structure of the telescopic rod 22 is composed of two or more rods connected together, and the telescopic function can be realized by changing the distance between the rods. These rods are kept stable through specific connection methods, such as screw connection, gear connection, etc. For example, in screw connection, by rotating the screw, the two rods can be gradually separated or approached, thereby changing the length of the entire rod.

[0085] As Figure 4As shown, the working distance of the second imaging device 21 is at the correct focusing imaging distance that maintains a horizontal distance D3 from the patient 6. The second imaging device 21 is installed on the telescopic rod 22. By adjusting the height in the direction perpendicular to the ground, the second imaging device 21 can adapt to the heights of different patients 6. Optionally, it is necessary to ensure a sufficient height D4 from the ground and a height D5 from the nose to ensure that the finger-to-nose test identification objects (the palm and nose of the patient 6) are within the frame. The patient 6 completes the finger-to-nose test to obtain different evaluation grades.

[0086] In some embodiments, second feature parameters are extracted from the second video signal, including: based on each frame image of the second video signal, extracting second image data covering a preselected area of the palm; annotating the positions of the palm joint points and the nose tip joint points based on the second image data; determining the distance between the palm joint points and the nose tip joint points based on the annotation results; when the distance is less than a set distance threshold, determining that the palm reaches the nose tip position; determining the time taken for the second specified test process as the difference between the time when the palm last reached the nose tip position and the time when the palm reaches the nose tip position this time; and judging the state of the palm based on the coordinate change of the palm joint points after the palm reaches the nose position, where the state includes whether there is tremor.

[0087] The palm joint points and the nose tip joint points can be key identification points preselected for analyzing and evaluating the rehabilitation status of hand function, such as Figure 5 the human pose estimation model shown. In a specific embodiment, the BlazePose 33-point model can be used to obtain the human pose estimation model based on the human image.

[0088] As an example, the position of the nose (nose tip) joint point is set as the target point, and it is considered that the specified nose position is reached within a certain radius of the target point. Optionally, when the distance between the palm joint point and the nose tip joint point is less than a preset distance, it is determined that the palm reaches the nose position.

[0089] In some embodiments, when it is confirmed that the palm reaches the nose position, the coordinate points of the palm joint points are read, and a certain number of coordinate points of the palm joints are collected in a container. First, stability filtering can be performed on the palm position, and the abscissa and ordinate of all coordinate points in the container are analyzed, and variance analysis is performed on their fluctuation degree. Those with fluctuations exceeding a certain degree can be determined to be trembling.

[0090] As an example, the specific calculation method of the fluctuation degree reachVibrate is shown in Equation (3), and finally the overall jitter situation is represented by the joint variance of the two coordinates.

[0091]

[0092] where, x i , y irespectively represent the abscissa and ordinate of the i-th data in the container, represents the mean value of x or y in this set of data, reachVibrate represents the degree of fluctuation, and n represents the number of data.

[0093] In some embodiments, the hand function rehabilitation evaluation system further includes a grip strength tester 3, as Figure 6 shown, the grip strength tester 3 includes a rod, and a plurality of high-resolution force sensors are installed on the rod. The patient 6 obtains different evaluation levels by pinching or gripping the force sensors. The plurality of force sensors ensure that the rod can not only distinguish the strength of pinching and gripping, but also distinguish the hand gesture state and strength of pinching and gripping.

[0094] In some embodiments, when using the hand function rehabilitation evaluation system, a suitable rehabilitation analysis model can be determined first. This model should cover various factors related to hand rehabilitation, such as muscle strength, joint stability, flexibility, etc., and be able to accurately reflect the patient's rehabilitation progress, while being maximally compatible with existing medical gold standards. In specific embodiments, considering comprehensively, the UFMA scale can be selected, as shown in Table 1. The UFMA scale is a scale used to evaluate upper limb functional movement activities. This evaluation form is often used in rehabilitation medicine, especially for patients with upper limb functional injuries or disorders. Through the UFMA evaluation form, medical professionals can objectively and quantitatively evaluate the upper limb function of patients, so as to formulate more accurate and effective rehabilitation plans.

[0095] Table 1 UFMA scale

[0096]

[0097] As can be seen from Table 1, among the 10 groups of measurement items, 3 items are related to joint stability (Item 1, Item 2, Item 8), where Items 1 and 2 examine the maximum and minimum angles, and Item 8 examines the jitter within the cycle. 5 items examine grip strength (mainly the items of grasping objects, Item 3, Item 4, Item 5, Item 6, Item 7), and certain resistance capabilities are examined according to different objects. 2 items examine coordination (Item 9, Item 10, where Item 9 examines the ability of the hand joint to reach the nose position, and Item 10 examines the time to reach the nose position).

[0098] Before performing hand function rehabilitation assessment, it is necessary to determine the corresponding reference baseline, i.e., the target parameter value, according to the characteristics and rehabilitation needs of different populations. Based on 8 healthy population categories of male juveniles, female juveniles, male youths, female youths, male middle-aged people, female middle-aged people, male elderly people, and female elderly people, the maximum and minimum extension lengths of each finger, the grip strength, the time taken to reach the nose position, the tremor condition of the hand when reaching the nose position, the full extension and full grip extension cycle will be measured respectively. The reference baselines obtained from these populations can be used as the average reference standard for assessment to help judge the rehabilitation progress of patients.

[0099] In some embodiments, the hand rehabilitation assessment model adopted by the hand function rehabilitation assessment system may include the following characteristic parameter information: the current extension length of each finger, whether the current palm reaches the nose position, the grip resistance condition, the tremor condition of the palm after reaching the nose position, the time taken to complete a specified cycle. Examples of the characteristic parameters of hand function rehabilitation are as shown in the following formula (4):

[0100]

[0101] Among them, X0 len represents the extension length of the thumb at the current acquisition moment, X1 len represents the extension length of the index finger at the current acquisition moment, X2 len represents the extension length of the middle finger at the current acquisition moment, X3 len represents the extension length of the ring finger at the current acquisition moment, X4 len represents the extension length of the little finger at the current acquisition moment, reachTarget represents whether the current palm reaches the nose position (such as near the nose), reachVibrate represents the tremor condition of the hand when the current palm reaches the nose position, reachTime represents the time taken since the last time reaching the nose position, roundTime represents the time taken since the last full extension or full grip, and force represents the grip strength.

[0102] In some other embodiments, a hand function rehabilitation assessment system is provided, as Figure 7 and Figure 8 shown, including: a hand evaluator 1, a hand activity instrument 7, a grip strength evaluator 3, and a main controller (not shown in the figure). Among them, the hand activity instrument 7 is configured to include a mechanical finger structure and allow the affected palm 61 of the patient 6 to wear. According to the output behavior value instruction, it drives the motor to drive the mechanical finger structure to perform corresponding bending actions and feedback the grip strength value of the hand activity instrument during the bending action of the mechanical finger.

[0103] The hand evaluator is configured to collect a first video signal of the healthy palm of the patient completing the first specified test process; the grip strength evaluator is configured to collect the grip strength value of the grip strength evaluator after the patient wears the hand activity instrument.

[0104] In the embodiment, by using the hand function rehabilitation evaluation system, the mirror pinch is realized based on the value iteration method. The basic idea is:

[0105] Synchronize the clock signals of the grip strength evaluator 3, the hand evaluator 1, and the hand activity instrument 7, that is, group the three data packets with the closest adjacent time differences into one group.

[0106] The hand evaluator 1 performs healthy-side movement, obtains the extension lengths of each finger, and updates the corresponding reward and punishment values of the hand evaluator according to the extension lengths of each finger.

[0107] According to the grip strength value of the grip strength evaluator feedback in real time by the grip strength evaluator 3, calculate the reward and punishment value of the grip strength evaluator, and calculate the value score of the grip strength evaluator. When the value score does not exceed the simple score threshold (that is, less than or equal to the value score threshold), update the reward and punishment value of the grip strength evaluator. The hand activity instrument 7 receives the output behavior value transmitted from the main controller (or PC).

[0108] The hand activity instrument 7 drives the mechanical finger structure according to the output behavior value of the previous round, and at the same time detects whether the grip strength value of the grip strength evaluator measured by the grip strength evaluator 3 meets the requirements. If it does not meet the requirements, extend the affected palm 61 of the patient 6 according to the current output behavior value, and obtain in real time the grip strength value of the hand activity instrument feedback during the extension process and the grip strength value of the grip strength evaluator feedback by the grip strength evaluator 3. Set the matrix of these two feedback values as the state value, and update each output behavior value in combination with the reward and punishment value of the hand evaluator and the reward and punishment value of the grip strength evaluator.

[0109] If the evaluation of the grip strength value of the grip strength evaluator reaches the standard, it means that the pinch is in place and the grip strength meets the requirements to exit the program and expand the hand activity instrument, waiting for the next pinch.

[0110] Use the grip strength evaluator 3 to obtain the feedback grip strength value of the grip strength evaluator in real time. If there is no reading, it means that there is no touch and pinch, and no record needs to be made.

[0111] Continue to refer to Figure 7 and Figure 8 , according to Figure 7 and Figure 8 The application method of the hand function rehabilitation evaluation system provided by the embodiment, as Figure 9 shown, includes:

[0112] Place the healthy hand 62 of patient 6 on the hand evaluator 1, and collect the first video signal of the patient's palm completing the first specified test process; use the main controller to extract the first image data covering the preselected area of the palm based on the first video signal, mark the positions of the palm joint points based on the first image data, confirm the extended lengths of each finger based on the marking results, and update the reward and punishment value of the hand evaluator according to the extended lengths of each finger;

[0113] Let the affected palm 61 of patient 6 wear the hand activity instrument 7, place the grip strength evaluator 3 inside the mechanical finger structure, and the mechanical finger structure does not contact the grip strength evaluator;

[0114] Use the grip strength evaluator 3 to obtain the grip strength value of the feedback grip strength evaluator in real time; calculate the reward and punishment value of the grip strength evaluator according to the grip strength value of the grip strength evaluator; if the value score of the grip strength evaluator does not exceed the value score threshold (i.e., less than or equal to the value score threshold), then update the reward and punishment value of the grip strength evaluator;

[0115] The hand activity instrument 7 obtains the output behavior value instruction of the previous round, drives the mechanical finger structure to hold the grip strength evaluator 3 according to the output behavior value instruction of the previous round, and uses the main controller to read the grip strength value of the feedback grip strength evaluator of the grip strength evaluator 3 and the grip strength value of the feedback hand activity instrument of the hand activity instrument;

[0116] Detect whether the corresponding grip strength value of the grip strength evaluator reaches the standard;

[0117] If it does not reach the standard, update the output behavior value instruction to drive the hand activity instrument according to the collected grip strength value of the grip strength evaluator, the grip strength value of the hand activity instrument, the target grip strength value of the grip strength evaluator, the target grip strength value of the hand activity instrument, and the current reward and punishment value of the hand evaluator and the reward and punishment value of the grip strength evaluator;

[0118] If it reaches the standard, the main controller outputs a reset instruction to drive the hand activity instrument to release the grip strength evaluator. As an example, the calculation method of the reward and punishment function of the hand evaluator is as follows, where p h (s′, r|s, a) is the reward and punishment value of the hand evaluator, last feedlength is the finger extended length feedback by the hand evaluator 1 in the previous round, target feedlength is the target finger extended length, last handforce is the grip strength value of the feedback grip strength evaluator of the grip strength evaluator 3 in the previous round, target handforce is the target grip strength value of the grip strength evaluator, times notfit is the total number of rounds entered for not meeting the grip strength standard.

[0119] p h (s′, r|s, a) = last feedlength - target feedlength +(last handforce-target handforce )*times notfit (5);

[0120] As an example, the calculation method of the reward and punishment function of the grip strength evaluator is as follows, where p g (s′, r|s, a) is the reward and punishment value of the grip strength evaluator, last feedforce is the grip strength value of the hand activity instrument 7 feedback in the previous round, target feedforce is the target grip strength value of the hand activity instrument, last handforce , target handforce , times notfit has the same definition as above.

[0121] p g (s′, r|s, a) = last feedforce -tareget feedforce +(last handforce -target handforce )*times notfit (5); As an example, the functional expression of the value score of the grip strength evaluator is shown in Equation (6).

[0122]

[0123] Among them, γ represents the weight of the current reward, noise represents random noise for exiting the local optimum, p g (s′, r|s, a) represents the reward and punishment value of the grip strength evaluator obtained in the previous round, V * (s) represents the current value score of the grip strength evaluator, V * (s′) represents the value score of the grip strength evaluator in the previous round.

[0124] The output action value is a matrix with one row and five columns, which can be understood as the extension amount of the motors that drive the five mechanical finger structures of the hand activity instrument 7 by the motors. The calculation formula of each output action value is shown in Equation (7) below.

[0125] action output = max finger (target feedforce *ratio feedforce +last handForce *ratio handforce +p g (s′,r|s,a)+p h (s′,r|s,a), finer maxdistance )(7);

[0126] Among them, actionoutput is the output behavior value, target feedforce is the grip strength value of the target hand activity instrument, last handforce is the grip strength value of the grip strength evaluator feedback by the previous round of grip strength evaluator 3, ratio feedforce is the coefficient of the grip strength value of the hand activity instrument, ratio handforce is the coefficient of the grip strength value of the grip strength evaluator, p h (s′, r|s, a) is the reward and punishment value of the current hand evaluator, previous round, p g (s′, r|s, a) represents the reward and punishment value of the grip strength evaluator obtained currently. The final output result is not allowed to exceed the maximum output length of the motor of the corresponding finger, finer maxdistance so as not to injure the fingers of the patient 6.

[0127] Figure 7 and Figure 8 In the rehabilitation scenario example shown, the healthy palm 62 of the patient 6 uses the hand evaluator 1 for system input, and the affected palm 61 wears the hand activity instrument 7 as shown Figure 7 and holds the grip rod of the grip strength evaluator 3 to perform grip strength evaluation. The patient 6 drives the hand activity instrument 7 to move through the input of the healthy palm 62, so that the grip strength evaluation of each pinching action reaches the required level, and then gradually makes the affected palm 61 approach the required grip strength level.

[0128] The hand function rehabilitation evaluation system and application method provided by the embodiments of the present application comprehensively use clinical evaluation scales and sensor data, can record and analyze hand movement activities more quickly and accurately, provide more comprehensive and objective patient rehabilitation progress information, and provide a basis for physicians to formulate personalized treatment plans. By recording and analyzing hand movement activity data, the whole process tracking and traceability of the rehabilitation progress can be realized, and it is ensured that the evaluation results are repeatable.

[0129] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be interpreted to include the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0130] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. Hand function rehabilitation evaluation system, characterized in that, It includes a hand evaluator and a main controller, and the hand evaluator and the main controller are communicatively connected; The hand evaluator is configured to collect a first video signal of a patient's palm completing a first specified test process; The main controller is configured to extract first image data covering a preselected area of the palm according to the first video signal, label the positions of the palm joint points based on the first image data, confirm first characteristic parameters of the palm completing the first specified test process based on the labeling results, compare the first characteristic parameters with corresponding first target parameter values, and obtain a hand function rehabilitation evaluation result; Wherein, the first specified test includes the palm moving from self-closure to opening, or the palm moving from self-opening to closure; the first characteristic parameters include the extended lengths of each finger, and the time interval between when the extended lengths of each finger reach the maximum reference value and when the extended lengths of each finger reach the minimum reference value; The system further includes a finger-to-nose test evaluator communicatively connected to the main controller, The finger-to-nose test evaluator is configured to collect a second video signal of a patient's palm completing a second specified test process; The main controller is configured to extract second characteristic parameters according to the second video signal; compare the second characteristic parameters with corresponding second target parameter values, and obtain a hand function rehabilitation evaluation result; Wherein, the second specified test includes a finger-to-nose test, and the second characteristic parameters include the time taken for the second specified test process and the state after the palm reaches the nose position, and the state includes whether there is tremor; When it is confirmed that the palm reaches the nose position, read the coordinate points of the palm joint points, collect the coordinate points of a certain number of palm joints in a container, analyze the abscissa and ordinate of all the coordinate points in the container, and perform variance analysis on their fluctuation degree. Those with fluctuations exceeding the set degree can be determined to have tremors. The specific calculation method of the fluctuation degree reachVibrate is shown in the following formula: Among them, x i , y i respectively represent the abscissa and ordinate of the i-th data in the container, represents the mean value of x or y in this set of data, reachVibrate represents the degree of fluctuation, and n represents the number of data.

2. The hand function rehabilitation evaluation system according to claim 1, wherein, The hand evaluator includes a first imaging device and a collection device body. The first imaging device is arranged in the collection device body, and the collection device body is provided with a collection window corresponding to the position of the first imaging device, so that the first imaging device can collect the first video signal through the collection window.

3. The hand function rehabilitation evaluation system according to claim 1, characterized in that The expression for determining the extended length of the palm based on the coordinates of the palm root and the coordinates of each fingertip is: Among them, X0 len represents the length of the thumb extended at the current acquisition moment, X1 len represents the length of the index finger extended at the current acquisition moment, X2 len represents the length of the middle finger extended at the current acquisition moment, X3 len represents the length of the ring finger extended at the current acquisition moment, X4 len represents the length of the little finger extended at the current acquisition moment, x00 and y00 are the abscissa and ordinate of the palm root, x0 n , y0 n respectively represent the abscissa and ordinate of the thumb tip at the current acquisition moment, x1 n , y1 n respectively represent the abscissa and ordinate of the index finger tip at the current acquisition moment, x2 n , y2 n respectively represent the abscissa and ordinate of the middle finger tip at the current acquisition moment, x3 n , y3 n respectively represent the abscissa and ordinate of the ring finger tip at the current acquisition moment, x4 n , y4 n represent the abscissa and ordinate of the little finger tip at the current acquisition moment; x0 r , y0 r respectively represent the change in abscissa and the change in ordinate of the thumb tip, x1 r , y1 r respectively represent the change in abscissa and the change in ordinate of the index finger tip at the current acquisition moment, x2 r , y2 r respectively represent the change in abscissa and the change in ordinate of the middle finger tip at the current acquisition moment, x3 r , y3 r respectively represent the change in abscissa and the change in ordinate of the ring finger tip at the current acquisition moment, x4 r , y4 r respectively represent the change in abscissa and the change in ordinate of the little finger tip at the current acquisition moment.

4. The hand function rehabilitation evaluation system according to claim 1, characterized in that, Determining the time interval between when the extended lengths of each finger reach the maximum reference value and when the extended lengths of each finger reach the minimum reference value includes: Presetting the maximum reference value and the minimum reference value of the extended length of each finger; Real-time detecting the extended length of each finger at a preset sampling rate; If there is a finger whose extended length exceeds the maximum reference value or the minimum reference value, update the maximum reference value and the minimum reference value; Record the first moment when the extended lengths of each finger reach the maximum reference value, and the second moment when the extended lengths of each finger reach the minimum reference value, and determine the difference between the first moment and the second moment as the time interval.

5. The hand function rehabilitation evaluation system according to claim 1, wherein, The finger-to-nose test evaluator includes a mobile platform, a telescopic rod, and a second imaging device. The telescopic rod is disposed on the mobile platform and can be telescoped in a direction perpendicular to the ground. The second imaging device is mounted on the telescopic rod.

6. The hand function rehabilitation evaluation system according to claim 1, wherein Extracting a second characteristic parameter according to the second video signal includes: Based on each frame image of the second video signal, extracting second image data covering a preselected area of the palm; based on the second image data, marking the positions of the palm joint points and the nose tip joint points; Based on the marking result, determining the distance between the palm joint point and the nose tip joint point; when the distance is less than a set distance threshold, determining that the palm reaches the nose tip position; determining the time used for the second specified test process as the difference between the time when the palm reached the nose tip position last time and the time when the palm reaches the nose tip position this time; and judging the state of the palm based on the coordinate change of the palm joint point after the palm reaches the nose position.

7. The hand function rehabilitation evaluation system according to claim 1, characterized in that, The system further includes a grip strength evaluator communicatively connected to the main controller, and the grip strength evaluator is configured to collect the grip strength value of the grip strength evaluator.

8. The hand function rehabilitation evaluation system according to any one of claims 1 to 7, characterized in that The system further includes: A hand activity instrument, configured to include a mechanical finger structure and be wearable, drive a motor according to an output behavior value instruction sent by the main controller to drive the mechanical finger structure to perform corresponding bending actions, and feedback the grip strength value of the hand activity instrument during the bending actions of the mechanical finger; A grip strength evaluator communicatively connected to the main controller; Wherein, the hand evaluator is configured to collect a first video signal of the healthy palm of the patient completing a first specified test process; The grip strength evaluator is configured to collect the grip strength value of the grip strength evaluator after the patient wears the hand activity instrument.

9. The application method of the hand function rehabilitation evaluation system according to claim 8, characterized in that, Including: Placing the healthy hand of the patient on the hand evaluator to collect a first video signal of the patient's palm completing a first specified test process; Using the main controller to extract first image data covering a preselected area of the palm according to the first video signal, marking the positions of the palm joint points based on the first image data, confirming the extended lengths of the fingers according to the extended lengths, and updating the reward and punishment value of the hand evaluator according to the reward and punishment function of the hand evaluator; Using the grip strength evaluator to obtain the feedback grip strength value of the grip strength evaluator in real time; calculating the reward and punishment value of the grip strength evaluator according to the grip strength value of the grip strength evaluator using the reward and punishment function of the grip strength evaluator; Calculating the value score of the grip strength evaluator, and if the value score of the grip strength evaluator is less than or equal to the value score threshold, updating the reward and punishment value of the grip strength evaluator; Making the affected palm of the patient wear the hand activity instrument, placing the grip strength evaluator inside the mechanical finger structure, and the mechanical finger structure is not in contact with the grip strength evaluator; The hand activity instrument obtains the output behavior value instruction of the previous round, drives the mechanical finger structure to hold the grip strength evaluator according to the output behavior value instruction, and uses the main controller to read the grip strength value of the grip strength evaluator feedback and the grip strength value of the hand activity instrument feedback by the hand activity instrument; Detecting whether the grip strength value of the grip strength evaluator reaches the standard; If the standard is not met, the output behavior value instruction is updated to drive the hand activity instrument according to the grip strength value of the grip strength evaluator obtained by collection, the grip strength value of the hand activity instrument, the grip strength value of the target grip strength evaluator, the grip strength value of the target hand activity instrument, and the current reward and punishment value of the hand evaluator and the reward and punishment value of the grip strength evaluator; If the standard is met, the main controller drives the hand activity instrument to release the grip strength evaluator.

Images