Hand function rehabilitation evaluation system and application method thereof

Abstract

The invention discloses a hand function rehabilitation assessment system and method, the system comprises a hand assessor and a main controller, the hand assessor collects a first video signal that the palm of a patient completes a first specified test process; the main controller obtains a first characteristic parameter according to the first video signal, and compares the first characteristic parameter with a corresponding first target parameter value to obtain a hand function rehabilitation evaluation result; wherein the first specified test comprises that the palm moves from closing to opening, or the palm moves from opening to closing; the first characteristic parameter comprises the extension length of each finger and the time interval between the time when the extension length of each finger reaches the maximum reference value and the time when the extension length of each finger reaches the minimum reference value. According to the embodiment of the invention, the hand evaluator collects the video signals of the palm of the patient completing the specified test process in real time, the signals can be processed in time, the feature parameters can be extracted, and thus the hand function rehabilitation evaluation result can be quickly obtained; through accurate comparison, the evaluation result is more objective and accurate.

Description

Technical Field

[0001] The present application belongs to the field of robot system control technology, and specifically relates to a hand function rehabilitation evaluation system and an application method thereof. Background Art

[0002] In the study of human limb movement, clinical and industrial evaluation methods have different methodological tendencies.

[0003] Limb rehabilitation assessment scales designed by clinical researchers usually design several groups of movement behaviors. While the patient completes the corresponding movement, the physician confirms the patient's rehabilitation rating through intuitive methods such as observation, questioning, and manual feeling. These methods have become the gold standard in related fields after years of trial and error by predecessors. However, these traditional hand function rehabilitation assessment methods are easily affected by the assessor's personal experience and skill level, resulting in inconsistency and subjectivity in the assessment results, and are often time-consuming and relatively cumbersome to operate.

[0004] In contrast, the evaluation methods in the industry are usually based on the sensor data of the designed observation system, which indirectly observes the signal changes generated by the patient's limb movements. These methods are innovative and the recording process is fast and convenient. However, due to the cross-disciplinary and latecomer advantages, the evaluation methods in the industry still need to be further improved, and there are also various methods for feature extraction and classification of evaluation results. Therefore, it is still a challenge to achieve the best results in terms of accuracy. Summary of the invention

[0005] The technical purpose of this application is to propose a hand function rehabilitation assessment system and its application method, so as to take into account the advantages of traditional clinical assessment methods and the efficiency of new industrial assessment methods, and improve the accuracy of hand rehabilitation assessment.

[0006] In order to achieve the above technical objectives, this application adopts the following technical solutions.

[0007] In a first aspect, an embodiment of the present application provides a hand function rehabilitation assessment system, comprising a hand evaluator and a main controller, wherein the hand evaluator and the main controller are communicatively connected;

[0008] The hand evaluator is configured to collect a first video signal of a patient's palm completing a first designated test process;

[0009] The main controller is configured to extract first image data covering a preselected area of ​​the palm according to the first video signal, mark the positions of the palm joints based on the first image data, confirm the first characteristic parameter of the palm completing the first specified test process based on the marking result, compare the first characteristic parameter with the corresponding first target parameter value, and obtain a hand function rehabilitation assessment result;

[0010] Among them, the first designated test includes the palm moving from closed to open, or the palm moving from open to closed; the first characteristic parameter includes the extended length of each finger, and the time interval between the extended length of each finger reaching the maximum reference value and the extended length of each finger reaching the minimum reference value.

[0011] In some embodiments, the hand evaluator includes a first camera device and a collection device body, the first camera device is arranged in the collection device body, and the collection device body is provided with a collection window corresponding to the position of the first camera device, so that the first camera device collects the first video signal through the collection window.

[0012] In some embodiments, the method for determining the extension length of each finger is as follows:

[0013] According to the labeling results, the ordinate and abscissa of each fingertip joint at the current acquisition time and the ordinate and abscissa of each fingertip joint at the previous acquisition time are obtained, and the change amount of the ordinate and abscissa of each fingertip joint is determined;

[0014] The extension length of each finger is determined according to the variation, and the expression is:

[0015]

[0016]

[0017] Among them, X0 len Indicates the length of the thumb at the current collection time, X1 len Indicates the length of the index finger at the current collection time, X2 len Indicates the length of the middle finger at the current acquisition time, X3 len Indicates the length of the ring finger at the current collection time, X4 len Indicates the length of the little finger at the current collection time, x0 0 ,y0 0 Respectively represent the horizontal and vertical coordinates of the thumb tip at the last acquisition moment, x1 0 ,y1 0 Respectively represent the horizontal and vertical coordinates of the index finger tip at the last acquisition moment, x2 0 ,y2 0 Respectively represent the horizontal and vertical coordinates of the tip of the middle finger at the last acquisition moment, x3 0 ,y3 0 Respectively represent the horizontal and vertical coordinates of the tip of the ring finger at the last acquisition time, x4 0 ,y4 0 Respectively represent the horizontal and vertical coordinates of the tip of the little finger at the last acquisition moment, x0 n ,y0n Respectively represent the horizontal and vertical coordinates of the thumb tip at the current acquisition time, x1 n ,y1 n Respectively represent the horizontal and vertical coordinates of the index finger tip at the current acquisition time, x2 n ,y2 n Respectively represent the horizontal and vertical coordinates of the middle finger tip at the current acquisition time, x3 n ,y3 n Respectively represent the horizontal and vertical coordinates of the ring finger tip at the current acquisition time, x4 n 、y4 n Indicates the horizontal and vertical coordinates of the tip of the little finger at the current acquisition time; x0 r ,y0 r Respectively represent the horizontal and vertical coordinate changes of the thumb tip, x1 r ,y1 r They represent the horizontal and vertical changes of the index finger tip at the current acquisition time, respectively, x2 r ,y2 r They represent the horizontal and vertical changes of the middle finger tip at the current acquisition time, respectively, x3 r ,y3 r Respectively represent the horizontal and vertical coordinate changes of the ring finger tip at the current acquisition time, x4 r ,y4 r They respectively represent the horizontal and vertical coordinate changes of the tip of the little finger at the current acquisition moment.

[0018] In some embodiments, determining the time interval between when the extended lengths of the fingers all reach the maximum reference value and when the extended lengths of the fingers all reach the minimum reference value includes:

[0019] Preset the maximum reference value and the minimum reference value of the extension length of each finger;

[0020] Detect the extension length of each finger in real time at a preset sampling rate;

[0021] If the extended length of a finger exceeds the maximum reference value or the minimum reference value, updating the maximum reference value and the minimum reference value;

[0022] The first moment when the extended length of each finger reaches the maximum reference value and the second moment when the extended length of each finger reaches the minimum reference value are recorded, and the difference between the first moment and the second moment is determined as the time interval.

[0023] In some embodiments, the system further comprises a finger-to-nose test evaluator communicatively connected to the main controller,

[0024] The finger-to-nose test evaluator is configured to collect a second video signal of the patient's palm completing a second designated test process;

[0025] The main controller is configured to extract a second characteristic parameter according to the second video signal; compare the second characteristic parameter with a corresponding second target parameter value to obtain a hand function rehabilitation assessment result;

[0026] The second designated test includes a finger-to-nose test, and the second characteristic parameter includes the time taken for the second designated test process and the state of the palm after reaching the nose position.

[0027] In some embodiments, the finger-to-nose test evaluator includes a mobile platform, a telescopic rod and a second camera device, wherein the telescopic rod is arranged on the mobile platform and can be extended and retracted in a direction perpendicular to the ground; and the second camera device is installed on the telescopic rod.

[0028] In some embodiments, extracting a second characteristic parameter according to the second video signal includes:

[0029] Based on each frame of the second video signal, extract second image data covering a preselected area of ​​the palm; and mark the positions of the palm joints and the nose tip joints based on the second image data;

[0030] Based on the labeling results, determine the distance between the palm joint point and the nose tip joint point; when the distance is less than the set distance threshold, determine that the palm has reached the nose tip position; determine the time taken for the second specified test process by the difference between the time when the palm last reached the nose tip position and the time when the palm reaches the nose tip position this time; judge the state of the palm based on the coordinate change of the palm joint point after the palm reaches the nose position, the state including whether it is tremor.

[0031] In some embodiments, the system further comprises a grip strength evaluator communicatively connected to the main controller, wherein the grip strength evaluator is configured to collect a grip strength value.

[0032] In a second aspect, the hand function rehabilitation assessment system provided by the embodiment of the present application based on the above embodiment further includes:

[0033] A hand activity instrument, which is configured to include a mechanical finger structure and can be worn, drives a motor according to an output behavior value instruction sent by the main controller to drive the mechanical finger structure to perform a corresponding bending action, and feeds back a hand activity instrument grip force value during the bending action of the mechanical finger;

[0034] a grip strength evaluator, communicatively connected to the main controller;

[0035] Wherein, the hand evaluator is configured to collect a first video signal of the patient's healthy palm completing a first designated test process;

[0036] The grip strength evaluator is configured to collect a grip strength value of the patient after the patient wears the hand activity instrument.

[0037] In a third aspect, the present application provides an application method of the hand function rehabilitation assessment system provided by any feasible implementation method of the second aspect, including:

[0038] Placing the healthy hand of the patient on the hand evaluator, and collecting a first video signal of the patient's palm completing a first designated test process;

[0039] Extracting first image data covering a preselected area of ​​the palm using a main controller according to the first video signal, marking the position of the palm joints based on the first image data, confirming the extension length of each finger based on the marking result, and updating the reward and punishment value of the hand evaluator according to the extension length;

[0040] The grip strength evaluator is used to obtain the grip strength value of the grip strength evaluator in real time; the reward and punishment value of the grip strength evaluator is calculated according to the grip strength value of the grip strength evaluator; and the value score of the grip strength evaluator is calculated. If the value score of the grip strength evaluator is less than or equal to the value score threshold, the reward and punishment value of the grip strength evaluator is updated;

[0041] The patient wears the hand activity instrument on the affected palm, and places a grip strength evaluator inside a mechanical finger structure, without the mechanical finger structure contacting the grip strength evaluator;

[0042] The hand activity instrument obtains the output behavior value instruction of the upper wheel, drives the mechanical finger structure to hold the grip strength evaluator according to the output behavior value instruction, uses the main controller to read the grip strength value of the grip strength evaluator fed back by the grip strength evaluator and the grip strength value of the hand activity instrument fed back by the hand activity instrument; and detects whether the grip strength value of the grip strength evaluator meets the standard;

[0043] If the standard is not reached, the output behavior value instruction is updated to drive the hand activity instrument according to the acquired grip strength value of the grip strength evaluator, the grip strength value of the hand activity instrument, the target grip strength value of the grip strength evaluator, the target grip strength value of the hand activity instrument, and the current reward and punishment value of the hand evaluator and the reward and punishment value of the grip strength evaluator;

[0044] If the standard is met, the main controller drives the hand activity instrument to release the grip strength evaluator.

[0045] Compared with the prior art, the hand function rehabilitation evaluation system provided in the embodiment of the present application collects video signals of the patient's palm completing the specified test process in real time through the hand evaluator, and the main controller can process these signals in real time and extract characteristic parameters, so as to quickly obtain the hand function rehabilitation evaluation results. This real-time performance helps to timely understand the patient's rehabilitation status and provide timely feedback for rehabilitation treatment; the characteristic parameters extracted by the video signal are accurately compared with the preset target parameter values, and the evaluation results are more objective and accurate. This method reduces the influence of subjective factors on the evaluation results and improves the credibility and consistency of the evaluation.

[0046] Compared with the prior art, the application method of the hand function rehabilitation assessment system provided in the embodiment of the present application has more objective and accurate assessment results. This method reduces the influence of subjective factors on the assessment results, improves the credibility and consistency of the assessment, and is conducive to the patient to drive the hand activity instrument to exercise through input through the healthy palm, so that the grip strength assessment of each pinching action reaches a level that meets the requirements, and then gradually makes the affected palm approach the grip strength level that meets the requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are for explanation purposes only and are not intended to limit the scope of the present application in any way. In addition, the shapes and proportional dimensions of the components in the drawings are only for illustration purposes and are used to help understand the present application. They do not specifically limit the shapes and proportional dimensions of the components of the present application. Under the guidance of the present application, those skilled in the art can select various possible shapes and proportional dimensions to implement the present application according to specific circumstances. In the drawings:

[0048] Figure 1 This is a structural block diagram of a hand evaluator in a hand function rehabilitation evaluation system in an embodiment of the present application;

[0049] Figure 2 This is a schematic diagram of the extended length of the palm in the embodiment of the present application;

[0050] Figure 3 A schematic diagram of the updating process of the maximum reference value and the minimum reference value in the embodiment of the present application;

[0051] Figure 4 This is a structural block diagram of a finger-to-nose test evaluator of a hand function rehabilitation evaluation system in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a human body posture estimation model used in another embodiment of the present application;

[0053] Figure 6 This is a structural block diagram of a grip strength tester of a hand function rehabilitation evaluation system in an embodiment of the present application;

[0054] Figure 7This is a structural side view block diagram of a hand function rehabilitation evaluation system in another embodiment of the present application;

[0055] Figure 8 It is a structural front view block diagram of a hand function rehabilitation evaluation system in another embodiment of the present application;

[0056] Fig. 9 A schematic diagram of the application method of the hand function rehabilitation assessment system provided in the embodiment of the present application;

[0057] Reference numerals:

[0058] 1-hand evaluator, 2-finger-nose test evaluator, 3-grip strength tester, 4-display, 5-palm, 6-patient, 7-hand activity instrument, 11-first camera device, 12-collection device body, 13-collection window, 21-second camera device, 22-telescopic rod, 23-support rod, 24-moving platform, 25-universal wheel, 61-affected palm, 62-healthy palm. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0060] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features.

[0061] At present, there is a set of guiding principles for the medical standards of hand rehabilitation assessment. These guiding principles are based on clinical practical experience and scientific research, are widely accepted, and are considered industry best practices. In hand rehabilitation assessment, medical standards usually include assessment scales and scoring systems for various limb movement behaviors. These scales and scoring systems have been clinically verified and revised for many years, and have been adopted in a wide range of clinical practices and have become the gold standard for evaluating the effectiveness of hand rehabilitation. The mainstream medical assessment scale benchmarks can generally be divided into muscle changes, movement patterns, and changes in hand function.

[0062] However, these traditional hand function rehabilitation assessment methods often rely on the experience and subjective judgment of doctors or therapists, and are easily affected by the assessor's personal experience and skill level, resulting in inconsistency and subjectivity in the assessment results. They are often time-consuming and the operation process is relatively cumbersome.

[0063] The hand function rehabilitation evaluation system provided in the embodiment of the present application collects video signals of the patient's palm completing the specified test process in real time through the hand evaluator, and the main controller can process these signals in time and extract characteristic parameters, so as to quickly obtain the hand function rehabilitation evaluation results. This real-time performance helps to timely understand the patient's rehabilitation status and provide timely feedback for rehabilitation treatment; the characteristic parameters extracted by the video signal are accurately compared with the preset target parameter values, and the evaluation results are more objective and accurate. This method reduces the influence of subjective factors on the evaluation results and improves the credibility and consistency of the evaluation.

[0064] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0065] An embodiment of the present application provides a hand function rehabilitation assessment system, including a hand evaluator and a main controller, wherein the hand evaluator and the main controller are communicatively connected; the hand evaluator is configured to collect a first video signal of a patient's palm completing a first specified test process; the main controller is configured to extract first image data covering a preselected area of ​​the palm according to the first video signal, mark the positions of the palm joints based on the first image data, confirm a first characteristic parameter of the palm completing the first specified test process based on the marking result, compare the first characteristic parameter with the corresponding first target parameter value, and obtain a hand function rehabilitation assessment result; wherein the first specified test is the palm moving from closed to open, or the palm moving from open to closed; the first characteristic parameter includes the extended length of each finger, and the time interval between the extended length of each finger reaching a maximum reference value and the extended length of each finger reaching a minimum reference value.

[0066] Communication connection refers to the communication between connected devices through the transmission and interaction of signals, completing the transmission and exchange of information. Communication connection can include wired connection and wireless connection. Wired connection is usually achieved through physical media such as cables and optical fibers, while wireless connection uses wireless signals such as electromagnetic waves for transmission.

[0067] In some embodiments, the structural block diagram of the hand evaluator 1 is as follows: Figure 1 As shown, the hand evaluator 1 includes a first camera device 11 and a collection device body 12. The first camera device 11 is placed in the collection device body 12. The collection device body 12 is provided with a collection window 13 corresponding to the position of the first camera device 11, so that the first camera device 11 can collect the second video signal through the collection window 13.

[0068] In some embodiments, the acquisition device body 12 is a sealed PVC (Polyvinyl chloride) box, and a transparent mounting cover may be provided on the top of the acquisition device body 12 to form a transparent acquisition window 23. The height of the PVC box is D1, and during use, the palm 5 and the acquisition window 23 formed by the transparent mounting cover maintain a use distance of D2, which ensures that the first camera device 11 can obtain a sufficient focusing distance, so that the main controller ( Figure 1 (not shown) provides correct imaging, and the patient obtains different assessment levels by opening and closing his palm 5.

[0069] In a specific embodiment, the main controller may be a device with data processing and analysis capabilities, such as a personal computer (PC), an industrial control computer, an embedded system, a server, a tablet computer, or a notebook.

[0070] In some embodiments, the video stream recorded by the first camera device 11 passes through the hand detector of the algorithm program in the main controller and is marked as a rough area of ​​interest covering the entire palm 5, and is transmitted backward to the detection annotator, in which an accurate hand rectangular frame is generated, which marks the precise range of the hand.

[0071] The frame image with the hand rectangle marked is further cropped to obtain a rough hand model analysis frame, which will be sent to the joint annotator for palm joint coordinate annotation. If the frame is too large, the region of interest will be annotated and then re-cropped. If the joint annotator data is successfully recognized, it will be annotated to generate a palm joint model. In the palm joint model with the palm joint polar coordinate system established at the base of the palm, the screen coordinates of all finger joints of the palm can be obtained.

[0072] In such Figure 2 In the example shown, only the coordinates of the base of the palm and the coordinates of each fingertip are actually needed to determine the extended length of the palm. R1, R2, and R3 are the extended lengths of the fingers of the palm joint model in this coordinate system. If a fist is made, the values ​​of R1, R2, and R3 will decrease accordingly. If the palm is stretched out, the values ​​of R1, R2, and R3 will increase accordingly.

[0073] The following is the basic structure of the finger extension length recognition result matrix. The matrix is ​​two-dimensional, with 5 rows and 2 columns. The basic row structure is 1 row and 2 columns. The first dimension stores the horizontal coordinate x of each fingertip (tip) joint in the original frame image, and the second dimension stores the vertical coordinate y of each fingertip joint in the original frame image.

[0074] For example, x0 n Indicates the latest horizontal coordinate of the thumb fingertip joint in the current refresh process, x1 nIndicates the latest horizontal coordinate of the index fingertip joint in the current refresh process, and then encodes each finger in turn; x0 o Indicates the latest horizontal coordinate of the thumb fingertip joint in the last round of refresh process, x1 o Indicates the latest horizontal coordinate of the index fingertip joint in the last round of refresh process, and then encodes each finger in turn. The encoding rule of the vertical coordinate can be based on the x-axis reasoning. The result matrix R is the distance between the fingertip joints of each finger and the two points in the current polar coordinate system. Then x0 r ,y0 r are the coordinate changes of the thumb fingertip joints respectively. It is easy to know that the radius R of the changed thumb is The following formulas (1) and (2) show how to calculate the extension length of each finger.

[0075]

[0076] In some embodiments, the first characteristic parameter also includes the time interval between when the extended length of each finger reaches the maximum reference value and when the extended length of each finger reaches the minimum reference value, that is, when the distal joint (fingertip joint) reaches the maximum reference value for the first time, that is, when the extended length of the fingers is the maximum, the timing starts, and when the fingertips reach the proximal joint (that is, the base of the palm), that is, when the extended length of the fingers reaches the minimum reference value, the timing stops, the time interval between the two timings is determined, and the time interval confirmed by the timing is compared with the preset time interval to obtain the hand function rehabilitation assessment result.

[0077] In some embodiments, Figure 3 As shown, determining the time interval between when a finger joint reaches a maximum reference value and when a finger joint reaches a minimum reference value includes: presetting a maximum reference value and a minimum reference value for the extension length of each finger in a reference coordinate system; detecting the extension length of each finger in real time at a preset sampling rate; if there is a finger whose extension length exceeds the maximum reference value or the minimum reference value, updating each maximum reference value and the minimum reference value; the finger extension length exceeding the maximum reference value or the minimum reference value can be used as the updated maximum reference value or the minimum reference value; recording a first moment when the extension length of each finger reaches the maximum reference value, and a second moment when the extension length of each finger reaches the minimum reference value, and determining the difference between the first moment and the second moment as the time interval.

[0078] The embodiment of the present application updates the maximum reference value and the minimum reference value in real time, can gradually adapt to and learn the user's finger movements, can adapt to the different rehabilitation stages of patients, and improve user experience and personalized services.

[0079] Some embodiments provide a hand function rehabilitation evaluation system, which also includes a finger-to-nose test evaluator communicatively connected to a main controller, the finger-to-nose test evaluator being configured to collect a second video signal of a patient's palm completing a second designated test process; the main controller being configured to extract a second characteristic parameter based on the second video signal; comparing the second characteristic parameter with a corresponding second target parameter value to obtain a hand function rehabilitation evaluation result; wherein the second designated test includes a finger-to-nose test, and the second characteristic parameter includes the time taken for the second designated test process and the state of the palm after reaching the nose position.

[0080] In the study of human limb movement, the finger-to-nose test is a commonly used clinical assessment method. In clinical assessment, the finger-to-nose test involves the patient being asked to straighten his or her forearm and touch the tip of his or her nose with the index finger, first slowly and then quickly, with eyes open and then closed, and repeating this action. Under normal circumstances, the finger-to-nose movement should be accurate and smooth. However, if there is a motor coordination disorder or muscle control problem, the patient's finger-to-nose movement may appear clumsy, inaccurate, uncoordinated, or unstable. However, this clinical assessment method is often time-consuming, costly to use, and the operation process is relatively cumbersome.

[0081] like Figure 4 As shown, in the hand function rehabilitation evaluation system provided by the embodiment of the present application, the finger-nose test evaluator 2 includes a mobile platform 24, a telescopic rod 22 and a second camera device 21. Optionally, a support rod 23 is provided on the mobile platform 14, and the support rod 23 is provided with a telescopic rod 22, and the telescopic rod 22 can be extended in a direction perpendicular to the ground; the second camera device 21 is installed on the telescopic rod 22. Universal wheels 25 are installed at the bottom of the mobile platform 24.

[0082] In some embodiments, the main controller is disposed on the mobile station 24 .

[0083] As an example, the hand function rehabilitation assessment system may further include a display 4 communicatively connected to the main controller, and the display 4 may be used to display data such as designated test prompts and test results.

[0084] In the embodiment, the telescopic rod 22 structure is composed of two or more rods connected together, and the telescopic function can be achieved by changing the distance between the rods. These rods are connected by a specific connection method, such as threaded connection, gear connection, etc., to maintain stability. For example, in a threaded connection, the two rods can be gradually separated or brought closer by rotating the thread, thereby changing the length of the entire rod.

[0085] like Figure 4As shown, the working distance of the second camera device 21 is at the correct focusing imaging distance of maintaining a horizontal distance D3 from the patient 6. The second camera device 21 is mounted on a telescopic rod 22, and the height is adjusted in a direction perpendicular to the ground so that the second camera device 21 can adapt to the heights of different patients 6. Optionally, it is necessary to ensure a sufficient height D4 from the ground and a height D5 from the nose to ensure that the finger-to-nose test identification object (the palm and nose of the patient 6) does not appear in the picture, and the patient 6 completes the finger-to-nose test to obtain different evaluation levels.

[0086] In some embodiments, extracting a second characteristic parameter based on a second video signal includes: extracting second image data covering a preselected area of ​​the palm based on each frame of the second video signal; marking the positions of the palm joints and the nose tip joints based on the second image data; determining the distance between the palm joints and the nose tip joints based on the marking results; when the distance is less than a set distance threshold, determining that the palm has reached the tip of the nose; determining the time required for the second specified test process by the difference between the time when the palm last reached the tip of the nose and the time when the palm this time reached the tip of the nose; and judging the state of the palm based on the change in coordinates of the palm joints after the palm reaches the nose position, the state including whether it is trembling.

[0087] The palm joints and nose tip joints can be pre-selected key identification points for analyzing and evaluating the rehabilitation status of hand function, such as Figure 5 The human body posture estimation model shown in the figure can use the BlazePose3 3-point model in a specific embodiment to obtain the human body posture estimation model based on the human body image.

[0088] As an example, the target point is set at the position of the nose (nose tip) joint point, and it is confirmed that the designated nose position has been reached within a certain radius of the target point. Optionally, when the distance between the palm joint point and the nose tip joint point is less than a preset distance, it is determined that the palm has reached the nose position.

[0089] In some embodiments, when it is confirmed that the palm has reached the nose position, the coordinate points of the palm joints are read, and a certain number of coordinate points of the palm joints are collected in a container. First, the palm position can be subjected to stability filtering, and the horizontal and vertical coordinates of all the coordinate points in the container are analyzed, and the degree of fluctuation is subjected to variance analysis. Fluctuations exceeding a certain degree can be determined as tremors.

[0090] As an example, the calculation method of the specific fluctuation degree reachVibrate is shown in formula (3), and the overall jitter situation is finally represented by the joint variance of the two coordinates.

[0091]

[0092] Among them, x i ,y iRespectively represent the horizontal and vertical coordinates of the i-th data in the container, It represents the mean of x or y in this set of data, reachVibrate represents the degree of fluctuation, and n represents the number of data.

[0093] In some embodiments, the hand function rehabilitation assessment system further comprises a grip strength tester 3, such as Figure 6 As shown, the grip strength tester 3 includes a rod with multiple high-resolution force sensors installed on it. The patient 6 obtains different evaluation levels through pinching or gripping the force sensors. The multiple force sensors ensure that the rod can not only distinguish the pinching strength, but also distinguish the hand posture and strength of the pinching.

[0094] In some embodiments, when using the hand function rehabilitation assessment system, a suitable rehabilitation analysis model can be determined first. The model should cover various factors related to hand rehabilitation, such as muscle strength, joint stability, flexibility, etc., and be able to accurately reflect the patient's rehabilitation progress, while being compatible with the existing medical gold standard to the greatest extent. In a specific embodiment, the UFMA scale can be selected after comprehensive consideration, as shown in Table 1. The UFMA scale is a scale for evaluating upper limb functional motor activities. This assessment form is often used in rehabilitation medicine, especially for patients with upper limb functional injuries or disorders. Through the UFMA assessment form, medical professionals can objectively and quantitatively evaluate the patient's upper limb function, thereby formulating a more accurate and effective rehabilitation plan.

[0095] Table 1 UFMA scale

[0096]

[0097] As can be seen from Table 1, among the 10 measurement items, there are 3 items related to joint stability (item 1, item 2, and item 8), of which 1 and 2 examine the maximum and minimum angles, and 8 examines the jitter within the cycle. There are 5 items to examine grip strength (mainly the grasping object items, item 3, item 4, item 5, item 6, and item 7), and a certain resistance capacity is examined according to different objects. There are 2 items to examine coordination (item 9 and item 10, of which 9 examines the ability of the hand joint to reach the nose position, and 10 examines the time to reach the nose position).

[0098] Before conducting hand function rehabilitation assessment, it is necessary to determine the corresponding reference baseline, i.e., the target parameter value, according to the characteristics and rehabilitation needs of different populations. The maximum and minimum extension lengths of each finger, grip strength, time taken to reach the nose, hand tremor when reaching the nose, and full extension and full grip extension cycle will be measured and established based on eight healthy population categories, including male teenagers, female teenagers, male youth, female youth, male middle-aged, female middle-aged, male elderly, and female elderly. The reference baselines obtained from these populations can be used as the average reference standard for evaluation to help judge the patient's rehabilitation progress.

[0099] In some embodiments, the hand rehabilitation assessment model used by the hand function rehabilitation assessment system may include the following characteristic parameter information: the current extension length of each finger, whether the current palm reaches the nose position, the grip resistance, the palm tremor after reaching the nose position, and the time to complete a specified cycle. The hand function rehabilitation assessment characteristic parameters are exemplified by the following formula (4):

[0100]

[0101] Among them, X0 len Indicates the length of the thumb at the current collection time, X1 len Indicates the length of the index finger at the current collection time, X2 len Indicates the length of the middle finger at the current acquisition time, X3 len Indicates the length of the ring finger at the current collection time, X4 len It indicates the length of the little finger at the current collection moment, reachTarget indicates whether the current palm has reached the nose position (such as near the nose), reachVibrate indicates the hand tremor when the current palm reaches the nose position, reachTime indicates the time taken to reach the nose position last time, roundTime indicates the time taken to fully extend or fully grip last time, and force indicates the grip strength.

[0102] In some other embodiments, a hand function rehabilitation assessment system is provided, such as Figure 7 and Figure 8 As shown, it includes: a hand evaluator 1, a hand activity meter 7, a grip strength evaluator 3 and a main controller (not shown in the figure). The hand activity meter 7 is configured to include a mechanical finger structure and allows the affected palm 61 of the patient 6 to wear it, and drives the motor according to the output behavior value instruction to drive the mechanical finger structure to perform a corresponding bending action, and feeds back the hand activity meter grip strength value during the bending action of the mechanical finger.

[0103] The hand evaluator is configured to collect a first video signal of the patient's healthy palm completing a first designated test process; the grip strength evaluator is configured to collect the grip strength value of the grip strength evaluator after the patient wears the hand activity instrument.

[0104] In the embodiment, the hand function rehabilitation assessment system is used to implement mirror pinching based on value iteration. The basic idea is:

[0105] The clock signals of the grip strength evaluator 3, the hand evaluator 1 and the hand activity meter 7 are frame synchronized, that is, three data packets with the closest time difference are grouped together.

[0106] The hand evaluator 1 performs healthy side movement to obtain the extension length of each finger, and updates the corresponding hand evaluator reward and punishment value according to the extension length of each finger.

[0107] According to the grip strength value of the grip strength evaluator fed back in real time by the grip strength evaluator 3, the grip strength evaluator reward and punishment value is calculated, and the grip strength evaluator value score is calculated. When the value score does not exceed the simple score threshold (i.e., less than or equal to the value score threshold), the grip strength evaluator reward and punishment value is updated. The hand activity instrument 7 receives the output behavior value transmitted by the main controller (or PC).

[0108] The hand activity instrument 7 drives the mechanical finger structure according to the output behavior value of the previous round, and at the same time detects whether the grip strength value of the grip strength evaluator measured by the grip strength evaluator 3 meets the requirements. If the requirements are not met, the affected palm 61 of the patient 6 is stretched according to the current output behavior value, and the hand activity instrument grip strength value fed back by the hand activity instrument 7 and the grip strength evaluator grip strength value fed back by the grip strength evaluator 3 during the stretching process are obtained in real time, and the matrix of these two feedback values ​​is set as the state value, and each output behavior value is updated in combination with the hand evaluator reward and punishment value and the grip strength evaluator reward and punishment value.

[0109] If the grip strength value of the grip strength evaluator meets the standard, it means that the pinch is in place and the grip strength is sufficient to exit the program and unfold the hand activity instrument, waiting for the next pinch.

[0110] The grip strength evaluator 3 is used to obtain the grip strength value of the feedback grip strength evaluator in real time. If there is no reading, it means that there is no touch and pinch, and no record is made.

[0111] Continue to refer Figure 7 and Figure 8 ,according to Figure 7 and Figure 8 The application method of the hand function rehabilitation assessment system provided in the embodiment is as follows: Fig. 9 As shown, including:

[0112] Place the healthy hand 62 of the patient 6 on the hand evaluator 1, and collect a first video signal of the patient's palm completing a first designated test process; extract first image data covering a preselected area of ​​the palm according to the first video signal using a main controller, mark the position of the palm joints based on the first image data, confirm the extension length of each finger based on the marking result, and update the reward and punishment value of the hand evaluator according to the extension length of each finger;

[0113] The affected palm 61 of the patient 6 is made to wear the hand activity instrument 7, and the grip strength evaluator 3 is placed inside the mechanical finger structure, and the mechanical finger structure is not in contact with the grip strength evaluator;

[0114] The grip strength evaluator 3 is used to obtain the grip strength value of the feedback grip strength evaluator in real time; the grip strength evaluator reward and punishment value is calculated according to the grip strength value of the grip strength evaluator; if the grip strength evaluator value score does not exceed the value score threshold (i.e., is less than or equal to the value score threshold), the grip strength evaluator reward and punishment value is updated;

[0115] The hand activity instrument 7 obtains the output behavior value instruction of the upper wheel, drives the mechanical finger structure to hold the grip strength evaluator 3 according to the output behavior value instruction of the upper wheel, and uses the main controller to read the grip strength value of the grip strength evaluator fed back by the grip strength evaluator 3 and the grip strength value of the hand activity instrument fed back by the hand activity instrument;

[0116] Check whether the grip strength value of the corresponding grip strength evaluator meets the standard;

[0117] If the target is not reached, the output behavior value instruction is updated to drive the hand activity meter according to the acquired grip strength value of the grip strength evaluator, the grip strength value of the hand activity meter, the target grip strength value of the grip strength evaluator, the target grip strength value of the hand activity meter, and the current reward and punishment value of the hand evaluator and the reward and punishment value of the grip strength evaluator;

[0118] If the target is met, the main controller outputs a reset instruction to drive the hand activity instrument to release the grip strength evaluator. As an example, the calculation method of the hand evaluator reward and punishment function is as follows, where p h (s′, r|s, a) is the reward and penalty value of the hand evaluator, last feedlength The finger extension length fed back by the hand evaluator 1 in the previous round, target feedlength The finger target extension length, last handforce is the grip strength value of the grip strength evaluator fed back by the grip strength evaluator 3 in the previous round, target handforce is the target grip strength evaluator grip strength value, times notfit The total number of rounds entered for which the grip strength criteria were not met.

[0119] p h (s′, r|s, a) = last feedlength -target feedlength +(last handforce-target handforce )*times notfit (5);

[0120] As an example, the calculation method of the grip strength evaluator reward and penalty function is as follows, where p g (s′, r|s, a) is the reward and punishment value of the grip strength evaluator, last feedforce is the hand activity instrument grip strength value fed back by the hand activity instrument 7 on the previous round, target feedforce is the target hand activity instrument grip strength value, last handforce 、target handforce 、times notfit Same as above definition.

[0121] p g (s′, r|s, a) = Last feedforce -target feedforce +(Last handforce -target handforce )*times notfit (5);

[0123] As an example, the functional expression of the grip strength evaluator value score is shown in formula (6).

[0124]

[0125] Where γ represents the weight of the current reward, noise represents the random noise used to exit the local optimum, and p g (s′, r|s, a) represents the reward and penalty value of the grip strength evaluator obtained in the previous round, V * (s) represents the current grip strength evaluator value score, V * (s′) represents the grip strength evaluator value score of the previous round.

[0126] The output behavior value is a matrix with one row and five columns, which can be understood as the extension amount of the motors driving the five mechanical finger structures in the hand activity instrument 7. The calculation formula of each output behavior value is shown in the following formula (7).

[0127] action output =max finger (target feedforce *ratio feedforce +last handForce *ratio handforce +p g (s′,r|s,a)+p h (s′,r|s,a), finer maxdistance ) (7);

[0129] Action output To output the behavior value, target feedforce is the target hand activity instrument grip strength value, last handforce is the grip strength value of the grip strength evaluator fed back by the grip strength evaluator 3 in the previous round, ratio feedforce is the hand activity instrument grip strength coefficient, ratio handforce is the grip strength coefficient of the grip strength evaluator, p h (s′, r|s, a) is the current hand evaluator reward and penalty value, last round, p g (s′, r|s, a) represents the current reward and penalty value of the grip strength evaluator. The final output result is not allowed to exceed the maximum output length of the motor of the corresponding finger. maxdistance , so as to avoid injuring patient 6's fingers.

[0130] Figure 7 and Figure 8 In the rehabilitation scenario example shown, the healthy hand 62 of the patient 6 uses the hand evaluator 1 for system input, and the affected hand 61 wears Figure 7 The hand activity instrument 7 shown in the figure holds the grip rod of the grip strength evaluator 3 to evaluate the grip strength, and the patient 6 drives the hand activity instrument 7 to move through the healthy palm 62, so that the grip strength evaluation of each pinching action reaches a level that meets the requirements, and then gradually makes the affected palm 61 approach the required grip strength level.

[0131] The hand function rehabilitation assessment system and application method provided in the embodiment of the present application, by using a combination of clinical assessment scales and sensor data, can record and analyze hand movement activities more quickly and accurately, provide more comprehensive and objective patient rehabilitation progress information, and provide a basis for physicians to formulate personalized treatment plans. By recording and analyzing hand movement activity data, the entire rehabilitation progress can be tracked and traced, and the evaluation results can be ensured to be repeatable.

[0132] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0133] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. Hand function rehabilitation assessment system, It is characterized in that It includes a hand evaluator and a main controller, wherein the hand evaluator is communicatively connected with the main controller; The hand evaluator is configured to collect a first video signal of a patient's palm completing a first designated test process; The main controller is configured to extract first image data covering a preselected area of ​​the palm according to the first video signal, mark the positions of the palm joints based on the first image data, confirm the first characteristic parameter of the palm completing the first specified test process based on the marking result, compare the first characteristic parameter with the corresponding first target parameter value, and obtain a hand function rehabilitation assessment result; Among them, the first designated test includes the palm moving from closed to open, or the palm moving from open to closed; the first characteristic parameter includes the extended length of each finger, and the time interval between the extended length of each finger reaching the maximum reference value and the extended length of each finger reaching the minimum reference value.

2. The hand function rehabilitation evaluation system according to claim 1, It is characterized in that The hand evaluator includes a first camera device and a collection device body, wherein the first camera device is arranged in the collection device body, and the collection device body is provided with a collection window corresponding to the position of the first camera device, so that the first camera device collects the first video signal through the collection window.

3. The hand function rehabilitation evaluation system according to claim 1, It is characterized in that The method for determining the extension length of each finger is as follows: According to the labeling results, the ordinate and abscissa of each fingertip joint at the current acquisition time and the ordinate and abscissa of each fingertip joint at the previous acquisition time are obtained, and the change amount of the ordinate and abscissa of each fingertip joint is determined; The extension length of each finger is determined according to the variation, and the expression is: Among them, X0 len Indicates the length of the thumb at the current collection time, X1 len Indicates the length of the index finger at the current collection time, X2 len Indicates the length of the middle finger at the current acquisition time, X3 len Indicates the length of the ring finger at the current collection time, X4 len Indicates the length of the little finger at the current collection time, x0 0 ,y0 0 Respectively represent the horizontal and vertical coordinates of the thumb tip at the last acquisition moment, x1 0 ,y1 0 Respectively represent the horizontal and vertical coordinates of the index finger tip at the last acquisition moment, x2 0 ,y2 0 Respectively represent the horizontal and vertical coordinates of the tip of the middle finger at the last acquisition moment, x3 0 ,y3 0 Respectively represent the horizontal and vertical coordinates of the tip of the ring finger at the last acquisition time, x4 0 、y4 0 Respectively represent the horizontal and vertical coordinates of the tip of the little finger at the last acquisition moment, x0 n ,y0 n Respectively represent the horizontal and vertical coordinates of the thumb tip at the current acquisition time, x1 n ,y1 n Respectively represent the horizontal and vertical coordinates of the index finger tip at the current acquisition time, x2 n ,y2 n Respectively represent the horizontal and vertical coordinates of the middle finger tip at the current acquisition time, x3 n ,y3 n Respectively represent the horizontal and vertical coordinates of the ring finger tip at the current acquisition time, x4 n ,y4 n Indicates the horizontal and vertical coordinates of the tip of the little finger at the current acquisition time; x0 r ,y0 r Respectively represent the horizontal and vertical coordinate changes of the thumb tip, x1 r ,y1 r They represent the horizontal and vertical changes of the index finger tip at the current acquisition time, respectively, x2 r ,y2 r They represent the horizontal and vertical changes of the middle finger tip at the current acquisition time, respectively, x3 r ,y3 r Respectively represent the horizontal and vertical coordinate changes of the ring finger tip at the current acquisition time, x4 r ,y4 r They respectively represent the horizontal and vertical coordinate changes of the tip of the little finger at the current acquisition moment.

4. The hand function rehabilitation evaluation system according to claim 1, It is characterized in that Determining the time interval between when the extended lengths of the fingers all reach the maximum reference value and when the extended lengths of the fingers all reach the minimum reference value, comprises: Preset the maximum reference value and the minimum reference value of the extension length of each finger; Detect the extension length of each finger in real time at a preset sampling rate; If the extended length of a finger exceeds the maximum reference value or the minimum reference value, updating the maximum reference value and the minimum reference value; The first moment when the extended length of each finger reaches the maximum reference value and the second moment when the extended length of each finger reaches the minimum reference value are recorded, and the difference between the first moment and the second moment is determined as the time interval.

5. The hand function rehabilitation evaluation system according to claim 1, It is characterized in that The system also includes a finger-to-nose test evaluator connected in communication with the main controller, The finger-to-nose test evaluator is configured to collect a second video signal of the patient's palm completing a second designated test process; The main controller is configured to extract a second characteristic parameter according to the second video signal; compare the second characteristic parameter with a corresponding second target parameter value to obtain a hand function rehabilitation assessment result; The second designated test includes a finger-to-nose test, and the second characteristic parameter includes the time taken for the second designated test process and the state of the palm after reaching the nose position.

6. The hand function rehabilitation evaluation system according to claim 5, It is characterized in that The finger-to-nose test evaluator comprises a mobile platform, a telescopic rod and a second camera device. The telescopic rod is arranged on the mobile platform and can be telescoped in a direction perpendicular to the ground. The second camera device is installed on the telescopic rod.

7. The hand function rehabilitation evaluation system according to claim 1, It is characterized in that Extracting a second characteristic parameter according to the second video signal includes: Based on each frame of the second video signal, extract second image data covering a preselected area of ​​the palm; and mark the positions of the palm joints and the nose tip joints based on the second image data; Based on the labeling results, determine the distance between the palm joint point and the nose tip joint point; when the distance is less than the set distance threshold, determine that the palm has reached the nose tip position; determine the time taken for the second specified test process by the difference between the time when the palm last reached the nose tip position and the time when the palm reaches the nose tip position this time; judge the state of the palm based on the coordinate change of the palm joint point after the palm reaches the nose position, the state including whether it is tremor.

8. The hand function rehabilitation evaluation system according to claim 1, It is characterized in that The system further includes a grip strength evaluator communicatively connected to the main controller, wherein the grip strength evaluator is configured to collect a grip strength value.

9. The hand function rehabilitation evaluation system according to any one of claims 1 to 7, It is characterized in that The system further comprises: A hand activity instrument, which is configured to include a mechanical finger structure and can be worn, drives a motor according to an output behavior value instruction sent by the main controller to drive the mechanical finger structure to perform a corresponding bending action, and feeds back a hand activity instrument grip force value during the bending action of the mechanical finger; a grip strength evaluator, communicatively connected to the main controller; Wherein, the hand evaluator is configured to collect a first video signal of the patient's healthy palm completing a first designated test process; The grip strength evaluator is configured to collect a grip strength value of the patient after the patient wears the hand activity instrument.

10. The application method of the hand function rehabilitation evaluation system according to claim 9, It is characterized in that include: Placing the healthy hand of the patient on the hand evaluator, and collecting a first video signal of the patient's palm completing a first designated test process; Extracting first image data covering a preselected area of ​​the palm using a main controller according to the first video signal, marking the position of the palm joints based on the first image data, confirming the extension length of each finger based on the marking result, and updating the reward and punishment value of the hand evaluator according to the extension length; The grip strength evaluator is used to obtain the grip strength value of the feedback grip strength evaluator in real time; Calculate the reward and punishment value of the grip strength assessor according to the grip strength value of the grip strength assessor; Calculate the grip strength evaluator value score, and if the grip strength evaluator value score is less than or equal to the value score threshold, update the grip strength evaluator reward and punishment value; The patient wears the hand activity instrument on the affected palm, and places a grip strength evaluator inside a mechanical finger structure, without the mechanical finger structure contacting the grip strength evaluator; The hand activity instrument obtains the output behavior value instruction of the upper wheel, drives the mechanical finger structure to hold the grip strength evaluator according to the output behavior value instruction, and uses the main controller to read the grip strength value of the grip strength evaluator fed back by the grip strength evaluator and the grip strength value of the hand activity instrument fed back by the hand activity instrument; Check whether the grip strength value of the grip strength evaluator meets the standard; If the standard is not reached, the output behavior value instruction is updated to drive the hand activity instrument according to the acquired grip strength value of the grip strength evaluator, the grip strength value of the hand activity instrument, the target grip strength value of the grip strength evaluator, the target grip strength value of the hand activity instrument, and the current reward and punishment value of the hand evaluator and the reward and punishment value of the grip strength evaluator; If the standard is met, the main controller drives the hand activity instrument to release the grip strength evaluator.

Images