Hand function intelligent evaluation system and evaluation method

Abstract

A hand function intelligent evaluation system and method, wherein the system comprises: an optical sensor, a system framework, a workstation and a display, the display is used to play a test demonstration video obtained from a cloud server under the control of the workstation, to guide the tester to show the motion state of the test hand according to the test gesture in the test demonstration video, the optical sensor is arranged on the system framework, and the hand motion image of the tester showing the joint activity state in the test process is obtained in real time from multiple perspectives, and the hand motion image is sent to the workstation, the workstation is used to obtain the joint angle parameter of the tester according to the hand motion image, and obtain the hand function evaluation result of the tester in this test according to the joint angle parameter and output the evaluation result, the evaluation system and method can improve the accuracy and intelligence of the evaluation, and the cost is low.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an intelligent assessment system and method for hand function. Background Technology

[0002] Hand movement measurement and assessment is an important part of clinical work. It is an important step and means for clinicians, especially hand surgeons, to understand hand diseases, determine the type and nature of injuries, and make clinical diagnoses. It is also the basis for subsequent applications such as functional assessment, efficacy evaluation, and work capacity assessment.

[0003] Current methods for examining hand movements require clinicians to use specialized measuring equipment for face-to-face, hands-on manual examination and measurement. The limitations of this technology lie in its insufficient intelligence and accuracy in assessment. Specifically: the accuracy of manual measurement depends on the experience and operational procedures of the measurement personnel, thus requiring rigorous training; manual measurement methods can only measure one joint at a time, making the process time-consuming when multiple joints need to be measured; and due to the multi-joint, multi-degree-of-freedom nature of the fingers, single-view visual images suffer from severe occlusion, failing to fully record the finger's movement and reducing the accuracy of hand function assessment. Summary of the Invention

[0004] This application provides an intelligent hand function assessment system and method, which can improve the intelligence and accuracy of hand function assessment.

[0005] One embodiment of this application provides a hand function intelligent assessment system, including:

[0006] Optical sensors, system frame, workstation, and display;

[0007] The workstation connects the optical sensor and the display, and is used to obtain test demonstration videos from a cloud server;

[0008] The display is used to play the test demonstration video for the tester under the control of the workstation, so as to guide the tester to demonstrate the movement state of each joint of the fingers of the test hand according to the test gestures in the test demonstration video;

[0009] The optical sensor is mounted on the system frame and is used to acquire, from multiple perspectives in real time, hand motion images of the test subject displaying the joint activity state during the test, and send the hand motion images to the workstation.

[0010] The workstation is also used to obtain the hand joint angle parameters of the test subject based on the hand motion images, and to obtain the hand function evaluation result of the test subject in this test based on the hand joint angle parameters, and to output the hand function evaluation result.

[0011] This application also provides a method for intelligent assessment of hand function, which uses the intelligent assessment system for hand function described above to perform intelligent assessment of hand function. The method includes:

[0012] The test demonstration video is obtained from the cloud server, and the display is controlled to play the test demonstration video for the tester, so as to guide the tester to demonstrate the joint movement of the fingers of the test hand according to the test gestures in the test demonstration video, wherein the cloud server obtains the test demonstration video based on the tester's personal information through a large language model;

[0013] The optical sensor acquires real-time images of the test subject's hand movements during the test, showing the joint activity state, from multiple perspectives.

[0014] The hand joint angle parameters of the test subject are obtained based on the hand motion images, and the hand function evaluation results of the test subject in this test are obtained and output based on the hand joint angle parameters.

[0015] As can be seen from the above embodiments of this application, the intelligent hand function assessment system of this application includes optical sensors, a system framework, a workstation, and a display. The workstation acquires a test demonstration video from a cloud server. Under the control of the workstation, the display plays the test demonstration video for the test subject, guiding the test subject to demonstrate the joint angles of the test hand according to the test gestures shown in the video. Multiple optical sensors mounted on the system framework acquire real-time images of the test subject's hand movements from multiple perspectives during the test. The workstation obtains the test subject's joint angle parameters based on these images and then calculates and outputs the hand function assessment result. This system can capture the test subject's hand movements from multiple angles, improving the accuracy of the test. It can also program, modularize, and automate the hand function assessment process, improving the intelligence and accuracy of the assessment. Furthermore, before the assessment, a personalized test demonstration video can be configured for the test subject using a large language model, further improving the accuracy and efficiency of the assessment. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of the structure of an intelligent hand function assessment system provided in an embodiment of this application;

[0018] Figure 2 A palmar plane diagram illustrating the calculation of the metacarpophalangeal joint angle of the index finger in a hand function intelligent assessment system provided in another embodiment of this application;

[0019] Figure 3 A schematic diagram of the structure of an intelligent hand function assessment system provided in another embodiment of this application;

[0020] Figure 4 A flowchart illustrating the implementation of an intelligent hand function assessment method provided in an embodiment of this application.

[0021] Figure 5 A flowchart illustrating the implementation of a hand function intelligent assessment method provided in another embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "axial", "radial", "circumferential", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0024] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a communication connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. The technical solutions of this application are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0025] See Figure 1 This application provides a hand function intelligent assessment system, which includes: multiple optical sensors 10, a system frame 20, a workstation 30 and a display 40.

[0026] The workstation 30 connects to the optical sensor 10 and the display 40 via wired, wireless WiFi, or Bluetooth.

[0027] The system frame 20 has a detachable structure. Specifically, the system frame 20 includes multiple detachable components, including multiple straight rods and multiple connectors for connecting the rods. Each straight rod and each connector can be assembled into a complete system frame 20. After being disassembled into straight rods and connectors, the volume is reduced and it can be placed in a bag for easy carrying.

[0028] Workstation 30 can be a desktop computer or laptop computer with image and other data processing capabilities, or it can be a mobile phone with image and other data processing capabilities.

[0029] The workstation 30 has a built-in memory containing at least three pre-set algorithms: hardware control, posture estimation, and motion state analysis. The workstation 30 also includes a radio frequency (RF) transceiver and a processor. The processor can retrieve test demonstration videos from a cloud server via the RF transceiver, use the hardware control algorithms to control the display 40 to play the demonstration videos, and control the optical sensor 10 to simultaneously capture images of the test subject's hand movements from multiple perspectives. The processor uses the posture estimation algorithm to obtain multiple hand motion images from these images, including images of the test hand displaying test gestures. It extracts spatial information of the finger key points corresponding to the test gestures from these hand motion images. The processor uses the motion state analysis algorithm to calculate relevant index parameters of finger function based on the finger key point spatial information, and calculates the final evaluation result of hand function based on the index parameters. Specifically, the evaluation result is obtained based on preset clinical hand surgery hand movement physical examination items and motor function assessment standards. The RF transceiver can transmit signals based on WIFI, 2G / 3G / 4G / 5G cellular network protocols, and other similar network transmission protocols.

[0030] Each optical sensor 10 is mounted on the system frame 20 and is used to acquire information on the tester's position, hand posture, and hand movement process during the test from multiple angles. Specifically, the optical sensor 10 is an RGB camera or a depth camera, which captures a three-dimensional image of the tester's hand, thereby obtaining the complete and accurate movement process of the test hand within the motion space when displaying the test gesture. Figure 1 Taking three optical sensors 10 as an example, they can be set at positions that capture images of the hand from three directions: palmar side (front of the hand), radial side (thumb side), and ulnar side (little finger side). Among them, the palmar side view has the least obstruction from the back side during finger movement, and can record more information; the radial side mainly observes information about the thumb, index finger, and middle finger; and the ulnar side mainly records information about the ring finger and little finger.

[0031] In another embodiment, the optical sensor 10 may be further equipped with four optical sensors in four directions, including the back of the hand, so that the image information in these four directions can record the movement state of the fingers from all directions.

[0032] The display 40 has a human-computer interaction interface. When the workstation 30 is a computer, the display 40 can be a screen that matches the calculator. When the workstation 30 is a mobile phone, the display 40 can be the screen of the mobile phone.

[0033] During the assessment, the test subject enters personal information via the input devices of the display 40, including a keyboard and microphone, following prompts from the display 40. This personal information may include the test subject's name, age, gender, occupation, medical history, and hand sensory function. The hand sensory function can also be described verbally by the test subject and obtained through the microphone of the display 40.

[0034] The display 40 sends this personal information to the workstation 30 to create a personal profile for the tester.

[0035] Workstation 30 plays a test demonstration video on monitor 40. This video includes pre-test preparation instructions to guide the test subject in adjusting their seating, orientation, and camera position to ensure smooth subsequent testing and evaluation. Family members and medical personnel can also assist the test subject in making these adjustments under the video's guidance. The test demonstration video can be preset on workstation 30 or a cloud server, and workstation 30 can retrieve it from the cloud server.

[0036] Understandably, due to individual differences among testers, there may be additional requirements for test demonstration videos for different groups of people. For example, for people recovering from surgery, the demonstrated actions or gestures need to avoid secondary injury; for people engaged in specific professions, specific actions or gestures need to be involved in the test (e.g., for programmers, specific actions or gestures such as wrist rotation and thumb pointing are required); for the elderly, the speed of the actions demonstrated in the test demonstration video needs to be slower than normal, and so on.

[0037] Optionally, in other embodiments of this application, a personalized test demonstration video can be configured for the tester. Specifically, the personal information may also include the test purpose (e.g., overall hand function loss test, index finger fine motor coordination test, etc.). The workstation 30 can send this personal information to the cloud server. The cloud server can obtain a structured video script matching the personal information using a Large Language Model (LLM), and generate a corresponding test demonstration video based on the structured video script.

[0038] The LLM analyzes the input personal information and selects appropriate test actions or gestures for the user based on the analysis results (e.g., postoperative patients should avoid excessive weight-bearing movements) and core test indicators (e.g., amplitude and duration of the action or gesture). It then generates a corresponding structured video script based on the test action (or gesture) and core test indicators. This structured video script may include, but is not limited to, a sequence of actions (or gestures) (e.g., Step 1: Slowly open all five fingers to their maximum extent and hold for 3 seconds; Step 2: Clench your fist and hold for 3 seconds, etc.) and shooting requirements (e.g., camera angle, action duration, demonstration speed, number of repetitions, narration prompts, etc.).

[0039] The aforementioned LLM can be built upon general-purpose large language models such as GPT-4 (OpenAI), Claude 2 (Anthropic), and PaLM 2 (Google), and trained using various personal information (such as age, gender, occupation, medical history, and hand sensory function) and their corresponding test gestures as training data. In this way, by configuring personalized test demonstration videos for different test subjects through LLM, the assessment can be more targeted and its efficiency improved.

[0040] The test demonstration video also includes the required movement process and precautions for the hand test. The tester can complete the test process according to the content of the video. Each optical sensor 10 captures the movement process of the tester's test hand displaying the joint activity state from its own perspective throughout the test process and sends the captured image data to the workstation 30.

[0041] Specifically, under the control of the workstation 30, the display 40 plays the test demonstration video for the test subject to guide the test subject to demonstrate the movement state of the test hand according to the test gestures in the test demonstration video. The movement state can be used to demonstrate the range of motion of each joint of the test hand's fingers.

[0042] Based on the test demonstration video, the tester places their hand or both hands within the system frame and autonomously performs multiple test gestures within the acquisition range of the optical sensor. These test gestures can obtain the range of motion of each finger joint, including the minimum and maximum range of motion. Specific test gestures may include, but are not limited to, the following:

[0043] First gesture: Extend straight outwards;

[0044] The test subject stretches all fingers out naturally to both sides in the coronal plane as much as possible;

[0045] This first gesture is used to collect the extension angle of all finger joints. If there is an overextension angle, it is also collected.

[0046] Second gesture: Thumb abducted radially;

[0047] The test subject extends the thumb as far as possible and abducts it on the radial side of the coronal plane, while simultaneously extending and adducting the other four fingers as far as possible in the coronal plane;

[0048] The second gesture is used to capture the radial abduction angle of the thumb.

[0049] Third gesture: Thumb against palm;

[0050] The tester extends the thumb outwards towards the palm and keeps it in the same sagittal plane as the index finger, while keeping the other fingers in a straight and adducted state in the coronal plane as much as possible.

[0051] This third gesture is used to measure the distance between the thumb and the palm.

[0052] Fourth gesture; thumb adduction;

[0053] The test subject tries to tuck the thumb inward on the palm surface while keeping the other fingers straight and tucked in the coronal plane.

[0054] This fourth gesture is used to measure the distance of thumb adduction.

[0055] Fifth gesture: Thumb flexion;

[0056] The test subject tries to bend the thumb toward the palm while keeping the other fingers straight and adducted in the coronal plane.

[0057] This fifth gesture is used to capture the flexion angles of the thumb's metacarpophalangeal and interphalangeal joints.

[0058] Sixth gesture: Adduct the thumb on the radial side;

[0059] The test subject tries to adduct the thumb to the radial side of the index finger, while keeping the other fingers as straight and adducted as possible in the coronal plane.

[0060] This sixth gesture is used to capture the radial adduction angle of the thumb.

[0061] Seventh gesture: Hook hand;

[0062] The test subject holds the thumb naturally on the radial side of the index finger, and tries to flex the distal and proximal interphalangeal joints of the other four fingers, while keeping the metacarpophalangeal joints in a naturally extended state.

[0063] This seventh gesture is used to collect the flexion angles of the distal joints of the index, middle, ring, and little fingers.

[0064] Eighth gesture: Thumbs up;

[0065] The test subject holds the thumb naturally on the radial side of the index finger, while forcefully flexing all the interphalangeal and metacarpophalangeal joints of the other four fingers;

[0066] The eighth gesture is used to collect the proximal joint flexion angle and metacarpophalangeal joint angle of the index, middle, ring, and little fingers.

[0067] By using multiple optical sensors mounted on the system framework to capture images of the test subject's hand from multiple angles when making the aforementioned measurable gestures, image information of joint angles can be easily obtained. This also reduces interference with the application of the pose estimation algorithm caused by fingers occluding each other during movement.

[0068] Workstation 40 processes the image data of the aforementioned test gestures to obtain multiple images of each measurable gesture made by the test subject. Analysis of these images yields the angle parameters of each finger joint in the test subject's hand, including the thumb metacarpophalangeal joint flexion angle, thumb metacarpophalangeal joint extension angle, thumb interphalangeal joint flexion angle, thumb interphalangeal joint extension angle, thumb radial abduction angle, thumb radial adduction angle, thumb opposition distance, thumb adduction distance, index finger metacarpophalangeal joint flexion angle, index finger metacarpophalangeal joint extension angle, index finger proximal interphalangeal joint flexion angle, index finger proximal interphalangeal joint extension angle, index finger distal interphalangeal joint flexion angle, and index finger distal interphalangeal joint extension angle. The test measures 32 parameters, including the flexion and extension angles of the middle finger's metacarpophalangeal joint, the flexion and extension angles of the proximal and distal interphalangeal joints of the middle finger, the flexion and extension angles of the ring finger's metacarpophalangeal joint, the flexion and extension angles of the proximal and distal interphalangeal joints of the ring finger, and the flexion and extension angles of the little finger's metacarpophalangeal joint, as well as the flexion and extension angles of the proximal and distal interphalangeal joints of the little finger. These 32 parameters directly reflect the activity of the test subject's finger joints.

[0069] Workstation 30 analyzes the image data according to the posture estimation algorithm. Specifically, the image data can be processed into hand image data. The hand image data is analyzed to obtain hand joint angle parameters that reflect the hand joint activity state of the test subject. Based on the hand joint angle parameters, the evaluation result of hand function is obtained.

[0070] Specifically, workstation 30 employs a preset pose estimation algorithm that can obtain the coordinates of 21 key points for each hand in the test subject's hand image in real time. These 21 key points include the wrist and 20 key points for each finger from the thumb to the little finger. The coordinates of each key point include normalized coordinates (coordinate range 0-1) and depth information (the distance of each key point relative to the wrist position).

[0071] This pose estimation algorithm can extract the coordinates of multiple initial key points for each hand from multiple hand images in a hand motion image. Then, the coordinates of these initial key points can be corrected using non-first-view images in the hand motion image other than the first-view image, resulting in accurate three-dimensional coordinates of the corrected key points.

[0072] In a specific example, the preset attitude estimation algorithm may be the MediaPipe Hand attitude estimation algorithm, but the attitude estimation algorithm in this application embodiment is not limited to MediaPipe Hand.

[0073] Furthermore, based on the three-dimensional coordinates of these 21 key points, the first joint angle and the second joint angle are calculated. The first joint includes all joints except the metacarpophalangeal joints of the index, middle, ring, and little fingers. The second joint includes all joints except the first joint, namely the metacarpophalangeal joints of the index, middle, ring, and little fingers. Both the first and second joint angles include an extension angle and a flexion angle.

[0074] Specifically, based on the vector inner product algorithm, the angle of the first joint of each finger of the test hand is calculated according to the three-dimensional coordinates of the corrected key point;

[0075] In the first joint, a reference joint corresponding to the second joint is determined. The reference joint is a joint on the same finger as the second joint. For example, if the second joint is the metacarpophalangeal joint of the index finger, then the reference joint is any joint on the index finger other than the metacarpophalangeal joint. The corrected three-dimensional coordinates of the reference joint are obtained. Based on the normal vector of the palm plane and the corrected three-dimensional coordinates of the key point of the reference joint, the three-dimensional coordinates of the projection point of the corrected key point of the reference joint on the palm plane are obtained. The line connecting the three-dimensional coordinates of the projection point and the three-dimensional coordinates of the corrected key point of the second joint, and the line connecting the three-dimensional coordinates of the corrected key point of the reference joint and the three-dimensional coordinates of the corrected key point of the second joint are calculated to obtain the angle of the second joint.

[0076] In one embodiment, the first joint angle is calculated:

[0077] Formula 1-1 for calculating the angle θ between two adjacent first joints of any finger is:

[0078]

[0079] Where A(x1,y1,z1), B(x2,y2,z2) and C(x3,y3,z3) are the three-dimensional spatial coordinates of the three key points on the finger being calculated. and Let represent the vectors of two adjacent phalanges on the finger, and the above formula is the dot product algorithm.

[0080] Calculate the angle of the second joint:

[0081] See Figure 2 Given that the key points W(x1,y1,z1) of the wrist joint, O(x2,y2,z2) of the index finger metacarpophalangeal joint, and Q(x3,y3,z3) of the little finger metacarpophalangeal joint form the palm plane, find the joint angle P(x0,y0,z0). P can be a key point of any proximal interphalangeal joint of the index, middle, ring, or little finger. Figure 2 Taking P on the index finger as an example, it is the reference joint corresponding to the key point of the metacarpophalangeal joint of the index finger in the first joint, P′(x p ,y p ,z p ) represents the projection of point P, which needs to be calculated, onto the palm plane.

[0082] Let the normal vector of the plane be... The general equation for the plane of the palm can then be expressed as:

[0083] Ax + By + Cz + D = 0 (1-2)

[0084] Where, point P(x0,y0,z0) and point P′(x p ,y p ,z p The straight line l formed by ) and If they are parallel, then the parametric equation of l can be expressed as:

[0085]

[0086] We can obtain:

[0087]

[0088] Substituting formula (1-4) into formula (1-2) and simplifying, we get:

[0089]

[0090] Substituting formula (1-5) into formula (1-4) yields P′(x). p ,y p ,z p ):

[0091]

[0092] Of the 21 key points, there are no key points at the base of the second to fifth metacarpal bones, only key points of a single wrist. The calculation method of the second joint angle in this embodiment can reduce the interference of the abduction / adduction movements of the index, middle, ring, and little fingers in the coronal plane on the calculation results of the second joint angle.

[0093] Furthermore, it is determined whether the joint is in a state of hyperextension. For example, in the first measuring posture mentioned above, the five fingers may be hyperextended when they are extended outward. Taking the straightening of each finger joint as 0°, in this embodiment of the application, the movement of the thumb from 0° to the radial side in the coronal plane is judged as hyperextension, and the movement of the other fingers from 0° to the dorsal side in the sagittal plane is judged as hyperextension.

[0094] The method for determining whether a joint is hyperextended is as follows: (The method uses the cross product and the right-hand screw rule.)

[0095] Let the key point of the joint to be determined be O(a0,b0), and the proximal key point and distal key point of point O be A(a1,b1) and B(a2,b2), respectively.

[0096]

[0097] For determining hyperextension of the thumb joint, a and b are taken as x and y coordinates respectively. According to the right-hand screw rule, when the left thumb joint r < 0, the current angle is determined to be hyperextension; when the right thumb joint r > 0, the current angle is determined to be hyperextension.

[0098] For determining hyperextension of the finger joints other than the thumb, a and b are taken as y-axis coordinates, respectively. According to the right-hand screw rule, when r>0, the current angle is determined to be hyperextension.

[0099] If joint hyperextension is present, the angle of hyperextension is treated as a negative value. For example, if the angle of joint hyperextension is 5°, it is recorded as -5°, which serves as the standard for confirming the degree of joint function loss.

[0100] Furthermore, the distance between thumbs opposing each other and the distance between thumbs adducting each other can be calculated as follows:

[0101] Let point A(x1,y1,z1) and point B(x2,y2,z2) be the two key points whose distance needs to be calculated. First, use formula (1-8) to calculate the pixel distance between the two target key points. Then, using a known real-world length standard reference in the system framework, calculate the ratio between the pixel distance and the real-world distance, and then use formula (1-9) to convert the pixel distance of the distance parameter into the real-world distance of the distance parameter.

[0102]

[0103] Calculate the joint angles of each finger in sequence using the method described above. The result should be a numerical value.

[0104] Furthermore, the workstation 30 or the server connected to the workstation 30 is equipped with a finger function assessment assignment module. The assignment module is pre-configured with a statistical table of functional loss of each finger joint. The statistical table includes the correspondence between different numerical ranges of the angles of each finger joint corresponding to the 32 parameters and the functional loss. The joint angle corresponds to the joint range of motion.

[0105] Specifically, the correspondence may include the correspondence between different numerical ranges corresponding to the following parameters and the degree of joint function loss of the fingers: thumb metacarpophalangeal joint extension, thumb metacarpophalangeal joint flexion, thumb interphalangeal joint extension, thumb interphalangeal joint flexion, four fingers metacarpophalangeal joint extension, four fingers metacarpophalangeal joint flexion, four fingers proximal interphalangeal joint extension, four fingers proximal interphalangeal joint flexion, four fingers distal interphalangeal joint extension, four fingers distal interphalangeal joint flexion, thumb antipallidation standardized distance, and thumb adduction standardized distance.

[0106] For example, the correspondence between different numerical ranges of the thumb metacarpophalangeal joint extension angle and the degree of functional loss of that joint can include:

[0107] If the range of the thumb metacarpophalangeal joint extension angle is less than 25, then the corresponding joint function loss is 0.

[0108] It should be noted that when the thumb metacarpophalangeal joint is hyperextended, the hyperextension angle is recorded as a negative value, and the corresponding angle value range is <25, and the corresponding joint function loss is 0.

[0109] The range of the thumb metacarpophalangeal joint extension angle is [25, 35), which corresponds to a joint function loss of 3%.

[0110] The range of the thumb metacarpophalangeal joint extension angle is [35, 45), which corresponds to a joint function loss of 7%.

[0111] The range of the thumb metacarpophalangeal joint extension angle is [45, 55), which corresponds to a joint function loss of 10%.

[0112] If the extension angle of the thumb metacarpophalangeal joint is greater than 55, the corresponding joint function loss is 14%.

[0113] The correspondences for the other fingers are set in the same way as the correspondence for the thumb's metacarpophalangeal joint extension. The specific numerical ranges and the degree of joint function loss (in percentage) are set according to the actual application.

[0114] Furthermore, the statistical table also includes unique IDs for the angles of each finger joint. For example, the ID for the extension angle of the thumb metacarpophalangeal joint is T_MP_E; the ID for the flexion angle of the thumb metacarpophalangeal joint is T_MP_F.

[0115] Workstation 30 calls the corresponding table, obtains the corresponding functional loss degree of each joint based on the calculated joint angle of each finger, calculates the functional loss degree of each finger based on the functional loss degree of each joint, and calculates the overall functional loss degree of the hand based on the functional loss degree of each finger.

[0116] Specifically, the processor of workstation 30 is equipped with modules corresponding to joint angles. In one example, see the table below for the correspondence between modules and joint angle IDs, joint angle numerical ranges, and joint function loss percentages. Each module performs its own calculations and obtains the numerical range of joint angles and the joint function loss percentage.

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] Specifically, the functional loss degree I of the thumb is calculated. Thumb The formula is as follows:

[0124] I MP =T MPE +T MPF

[0125] I IP =T IPE +T IPE

[0126] I OP =T OP

[0127] I AD =T AD

[0128] I Thumb =(I MP +I IP ×(1-I MP ))×0.2+I OP +I AD

[0129] Among them, I MP T represents the degree of functional loss of the thumb metacarpophalangeal joint. MPE T represents the degree of loss of extension function of the thumb metacarpophalangeal joint. MPF The degree of loss of flexion function at the thumb metacarpophalangeal joint;

[0130] I IP T represents the degree of functional loss of the thumb interphalangeal joint. IPE T represents the degree of loss of extension function of the thumb interphalangeal joint. IPF The degree of loss of flexion function at the thumb metacarpophalangeal joint;

[0131] I OP T represents the degree of loss of thumb opposition function. OP The loss of the thumb-to-palm standardized distance;

[0132] I AD T represents the degree of thumb adduction function loss. AD The normalized distance loss for thumb adduction.

[0133] Calculate the functional loss of the four fingers other than the thumb (I) Finger The formula is as follows:

[0134] I MCP =F MCPE +F MCPF

[0135] I PIP =F PIPE +F PIPF

[0136] I DIP =F DIPE +F DIPF

[0137] I Index =I Middle =I Ring =I Little =I Finger

[0138] I Finger =I MCP +I PIP ×(1-I MCP )+I DIP ×[1-I MCP +I PIP ×(1-I MCP )]

[0139] I MCP F represents the degree of functional loss of the metacarpophalangeal joints of the four fingers. MCPE F represents the degree of loss of extension function of the metacarpophalangeal joints of the four fingers.MCPF The degree of loss of flexion function of the metacarpophalangeal joints of the four fingers;

[0140] I PIP F represents the degree of functional loss of the proximal interphalangeal joints of the four fingers. PIPE F represents the degree of loss of extension function of the proximal interphalangeal joints of the four fingers. PIPF The degree of loss of flexion function at the proximal interphalangeal joints of the four fingers;

[0141] I DIP F represents the degree of functional loss of the distal interphalangeal joints of the four fingers. DIPE F represents the degree of loss of extension function at the distal interphalangeal joints of the four fingers. DIPF The degree of loss of flexion function at the distal interphalangeal joints of the four fingers;

[0142] I Index I represents the loss of function of the indicator. Middle The middle finger functional loss; I Ring The index finger represents the loss of function; I Little The degree of functional loss in the little finger.

[0143] Furthermore, workstation 30 calculates the overall functional loss of the hand based on the functional loss of each finger and outputs it as the evaluation result. The formula for calculating the overall functional loss I of the hand is:

[0144] I = I Thumb ×0.4+I Index ×0.2+I Middle ×0.2+I Ring ×0.1+I Little ×0.1

[0145] Workstation 30 outputs the assessment results of the test subject's hand function. These results include the calculated overall functional loss of the hand, as well as abnormal joint movement parameters and corresponding images obtained based on this overall functional loss, along with information such as possible diseases and medical advice. Clinicians or rehabilitation physicians can obtain these assessment results to intuitively understand the test subject's hand movement disorder, assisting in subsequent targeted physical examinations. The overall functional loss of the hand can serve as a baseline for comparing subsequent treatment and rehabilitation stages, objectively analyzing the effectiveness of the disease / treatment. Abnormal movement images can serve as objective comparative evidence of intervention measures and as unified follow-up data.

[0146] The workstation 30 also outputs the tester's joint angle parameters, namely 32 joint angle parameters of the hand in the hand image corresponding to the measurement posture, which are used to directly reflect the activity of the tester's finger joints.

[0147] It outputs a hand image and the spatial coordinates of 21 key points in the corresponding image, mainly used to establish a hand image database and improve subsequent pose estimation algorithms.

[0148] Optionally, in other embodiments of this application, workstation 30 may also send the hand function assessment result to a designated terminal, such as a terminal device of a designated medical professional, so that the designated medical professional can review the hand function assessment result. Workstation 30 obtains the reviewed hand function assessment result returned by the terminal device and displays it on a monitor. Further, workstation 30 associates the reviewed hand function assessment with the test subject's personal information and the hand movement images of the test subject obtained during the test, and sends them to a cloud server for the retraining of the aforementioned LLM.

[0149] Optionally, in other embodiments of this application, such as Figure 3 As shown, the system also includes a cloud server 50. To improve the efficiency of data storage and processing, the cloud server 50 can be a distributed server cluster. This distributed server cluster includes: a distribution server 51, a data storage server 52, a demonstration video generation server 53, and a rehabilitation plan generation server 54.

[0150] The distribution server 51 stores the personal information sent by the workstation 30, the hand function assessment results, and the reviewed hand function assessment results in the data storage server 52. When it receives the personal information sent by the workstation 30, it instructs the demonstration video generation server 53 to retrieve the personal information from the data storage server 52 and generate a personalized test demonstration video based on the personal information using the LLM mentioned above.

[0151] The demonstration video generation server 53 is used to generate test demonstration videos according to the instructions of the distribution server 51 and store them in the data storage server 52, and to notify the distribution server 51 of the storage location of the generated test demonstration videos in the data storage server 52. Furthermore, the demonstration video generation server 53 is also used to periodically retrain the LLM using the data stored in the data storage server 52 to improve the accuracy of the LLM output results.

[0152] The distribution server 51 is also used to send a notification to the demonstration video generation server 53, instructing the workstation 30 to download the generated test demonstration video from the data storage server 52 according to the storage location of the notification from the demonstration video generation server 53.

[0153] The distribution server 51 is also used to instruct the rehabilitation plan generation server 54 to obtain the aforementioned hand function assessment results (or the reviewed hand function assessment results) and their associated personal information from the data storage server 52, and generate a personalized rehabilitation plan accordingly.

[0154] The rehabilitation plan generation server 54 is configured with an LLM (hereinafter referred to as the second LLM) for generating personalized rehabilitation plans for test subjects. The second LLM is similar in type to the LLM (or first LLM) used to generate structured video scripts, but it is trained using various personal information, hand function assessment results or reviewed hand function assessment results, and rehabilitation plans as training data. The rehabilitation plan generation server 54 uses the second LLM to determine whether the test subject needs a rehabilitation plan, generates a corresponding personalized rehabilitation plan for test subjects who need one, and stores it on the data storage server 52. Then, it sends the storage location of the generated personalized rehabilitation plan to the distribution server 51, so that the distribution server 51 instructs the workstation 30 to download the personalized rehabilitation plan from the data storage server 52 according to that storage location. In this way, using an LLM to generate personalized rehabilitation plans allows for a deep integration of professional medical knowledge and individual user differences, thereby solving problems such as strong generality, insufficient personalization, and delayed adjustments in traditional rehabilitation plans.

[0155] After downloading the personalized rehabilitation plan, workstation 30 can display it on a monitor or send it to the designated terminal for review by designated medical personnel. Workstation 30 receives the reviewed personalized rehabilitation plan from the designated terminal and displays it on the monitor. Further, workstation 30 sends the reviewed personalized rehabilitation plan to distribution server 51, which stores it in data storage server 52. This allows rehabilitation plan generation server 54 to periodically retrain the second LLM using the data stored in data storage server 52, thereby improving the accuracy of the second LLM output.

[0156] In this embodiment, the intelligent hand function assessment system includes an optical sensor, a system frame, a workstation, and a display. Under the control of the workstation, the display plays a test demonstration video for the tester, guiding the tester to demonstrate the joint movement status of the fingers of the test hand according to the test gestures in the video. This improves the ease of operation for the tester and increases the success rate of the test. The optical sensor, mounted on the system frame, acquires real-time images of the tester's hand movements from multiple perspectives, improving the comprehensiveness and accuracy of the acquired images. The workstation obtains the joint angle parameters of the tester's hand based on the hand movement images, and then obtains the hand function assessment result for this test based on these parameters. This improves the intelligence and accuracy of the hand function assessment, reduces the reliance on and inconvenience of manual measurement, and lowers the hardware and labor costs of the assessment.

[0157] The intelligent hand function assessment system is a standard multi-view intelligent physical examination and analysis equipment for hand movement (all-in-one machine): it consists of a standard optical sensor, system frame, workstation and display, and is used for autonomous testing or auxiliary case data collection in medical settings.

[0158] The mobile phone's optical sensor can be used as a data acquisition device. The test subject or a family member can hold the mobile phone and take a picture from the main perspective, abandoning the correction of other perspective data, and use it as a tool for home assessment and remote follow-up of the test subject.

[0159] The assessment process of hand function can be standardized, proceduralized, modularized, and automated, which can alleviate the current shortage of clinical hand surgery medical talents. It can partially meet the rehabilitation needs of patients with various diseases that have sequelae affecting hand function, especially patients living in economically and health-inadequate areas. It can reduce misdiagnosis, missed diagnosis, and complications caused by the lack of access to medical resources during the progression of their diseases, and save on medical costs to a certain extent.

[0160] By automating the relatively repetitive and mechanical labor of manual measurement and assessment, while improving the accuracy of the assessment, this invention can reduce the workload and time of medical staff in the assessment work, thus helping to alleviate their workload. At the same time, for primary healthcare institutions that do not have positions for hand surgeons and rehabilitation physicians, this invention can supplement the rehabilitation assessment and health monitoring functions of medical institutions.

[0161] Using standardized reference indicators to assess the hand motor function of test subjects can reduce inter-individual differences caused by varying levels of expertise among rehabilitation physicians, as well as differences in pre- and post-diagnosis and functional assessments conducted by the same physician. This can improve the accuracy and consistency of patient rehabilitation assessments and follow-ups.

[0162] The optical sensor is contactless and extremely small, making it easy to use and improving the user experience during testing. Users simply stand in the pre-designed testing area and follow the pre-programmed steps to quickly obtain clinical diagnostic and functional assessment results, simplifying the testing process and reducing its difficulty. Assessors require only simple training to complete the experiments and data processing, facilitating systematic application. Its convenience improves user compliance, promotes hand function rehabilitation, and enhances functional improvement. It also benefits primary healthcare education, increasing awareness and treatment rates for hand-related diseases.

[0163] Compared with large rehabilitation assessment equipment, the intelligent hand function assessment system in this application has advantages such as ease of use, low cost, small size and portability, and an acceptable appearance, and can serve as an effective supplement and partial replacement for large equipment.

[0164] This intelligent hand function assessment system can be integrated with internet hospitals to form an online rehabilitation diagnosis system. After a remote expert specifies the rehabilitation assessment content, the device guides the test subject to complete the assessment, and the results are fed back to the remote expert. This solves the problem that remote clinicians cannot obtain accurate motor data and functional status of test subjects in a timely and effective manner, thus hindering the provision of targeted follow-up treatment suggestions. The combined application of internet hospitals and intelligent devices will effectively improve the uneven distribution of rehabilitation resources.

[0165] The various data collected are essential for the development of subsequent clinical hand surgery hand multi-angle image databases and clinical hand pose estimation algorithms.

[0166] See Figure 4 This application also provides an intelligent assessment method for hand function, which assesses hand sensory function using the aforementioned intelligent hand function assessment system. The method may include the following steps:

[0167] S401. Play a test demonstration video for the tester via a monitor;

[0168] The workstation plays the video to the tester via a monitor to guide the tester to demonstrate the movement of the joints of the fingers of the test hand by following the test demonstration gestures in the video.

[0169] S402. The test subject's hand movement images during the test are acquired in real time from multiple perspectives using an optical sensor, showing the joint activity status of the hand.

[0170] The optical sensor sends the hand movement image to the workstation in real time.

[0171] S403. Obtain the hand joint angle parameters of the test subject based on the hand motion image, and obtain the hand function assessment result of the test subject in this test based on the hand joint angle parameters.

[0172] Output the hand function assessment results.

[0173] Specifically, the workstation uses a preset pose estimation algorithm to extract the coordinates of multiple initial key points from multiple hand images in the main view image of the hand motion image, and corrects the coordinates of multiple initial key points through non-main view images in the hand motion image to obtain the corrected three-dimensional coordinates of the key points.

[0174] Based on the three-dimensional coordinates of the corrected key points, the first joint angle and the second joint angle of the test hand in the hand image are calculated. The first joint includes all joints except the metacarpophalangeal joints of the index, middle, ring, and little fingers, and the second joint includes the metacarpophalangeal joints of the index, middle, ring, and little fingers.

[0175] Based on the vector inner product algorithm, the angle of the first joint of each finger of the test hand is calculated according to the three-dimensional coordinates of the corrected key point.

[0176] In the first joint, a reference joint corresponding to the second joint is determined, and the corrected three-dimensional coordinates of the reference joint are obtained;

[0177] Based on the normal vector of the palm plane and the three-dimensional coordinates of the key point after the reference joint correction, the three-dimensional coordinates of the projection point of the three-dimensional coordinates of the key point after the reference joint correction on the palm plane are obtained.

[0178] The angle of the second joint is obtained by connecting the three-dimensional coordinates of the projected point with the three-dimensional coordinates of the corrected key point of the second joint, and connecting the three-dimensional coordinates of the corrected key point of the reference joint with the three-dimensional coordinates of the corrected key point of the second joint.

[0179] The test subject's finger joints are determined using the vector outer product algorithm and the right-hand screw rule. If hyperextension is found, the angle of hyperextension is recorded as the negative value corresponding to that angle.

[0180] Calculate the distance between the thumbs facing each other and the distance between the thumbs adducting.

[0181] Based on the calculated angles of the first joint, the second joint, the thumb-to-palm distance, the thumb-adduction distance, and the joint hyperextension angle of each finger, the functional loss of each joint is obtained. Based on the functional loss of each joint, the functional loss of each finger is calculated, and the functional loss of the hand as a whole is calculated based on the functional loss of each finger. The functional loss of the hand as a whole is then output as the evaluation result.

[0182] Output the joint angle parameters of the test subject, as well as the hand image corresponding to the measurement posture in the hand motion image and the spatial coordinates of each key point in the hand image.

[0183] Optional, such as Figure 5 As shown, in other embodiments of this application, before step S401, the method further includes:

[0184] The S400 retrieves test demonstration videos from a cloud server, which uses the first major language model to obtain the test demonstration videos based on the tester's personal information.

[0185] Following step S403, the method further includes:

[0186] S404. Send the hand function assessment results to the cloud server so that the cloud server can obtain a corresponding personalized rehabilitation plan based on the hand function assessment results through the second language model.

[0187] S405 receives personalized rehabilitation plans sent from the cloud server and displays them to the test subject on a monitor.

[0188] For details regarding the specific testing process and other aspects not covered in this embodiment, please refer to the detailed description of the aforementioned intelligent hand function assessment system, which will not be repeated here.

[0189] In this embodiment, a hand function intelligent assessment system plays a test demonstration video for the tester, guiding the tester to demonstrate the joint activity of the fingers of the test hand according to the test gestures in the video. This improves the ease of operation for the tester and increases the success rate of the test. The system uses optical sensors mounted on the system frame to acquire hand motion images of the tester demonstrating joint activity from multiple perspectives in real time, improving the comprehensiveness and accuracy of the acquired images. The workstation obtains the tester's hand joint angle parameters based on the hand motion images, and obtains the hand function assessment results for this test based on the hand joint angle parameters. This improves the intelligence and accuracy of hand function assessment, reduces the reliance on and inconvenience of manual measurement, and reduces the hardware and labor costs of the assessment.

[0190] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0191] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0193] The above is a description of the intelligent hand function assessment system and assessment method provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A hand function intelligent assessment system, characterized in that, include: Optical sensors, system frame, workstation, and display; The workstation connects the optical sensor and the display, and is used to obtain test demonstration videos from a cloud server; The display is used to play the test demonstration video for the tester under the control of the workstation, so as to guide the tester to demonstrate the movement state of each joint of the fingers of the test hand according to the test gestures in the test demonstration video; The optical sensor is mounted on the system frame and is used to acquire, from multiple perspectives in real time, hand motion images of the test subject displaying the activity state of each joint during the test, and send the hand motion images to the workstation. The workstation is also used to obtain the hand joint angle parameters of the test subject based on the hand motion images, and to obtain the hand function evaluation result of the test subject in this test based on the hand joint angle parameters and output the hand function evaluation result. The workstation is also used to extract multiple initial key point coordinates from multiple hand images from the main view image in the hand motion image using a preset pose estimation algorithm, and to correct the multiple initial key point coordinates through the non-main view image in the hand motion image to obtain the corrected three-dimensional coordinates of the key points. Each hand includes 21 key points, which include 20 key points for each finger from the thumb to the little finger and 1 key point for the wrist. The 21 key points do not include the key points at the base of the second to fifth metacarpal bones. Each initial key point coordinate includes normalized coordinates and the distance of each key point relative to the wrist. The workstation is also used to calculate the first joint angle and the second joint angle of the test hand in the hand image based on the three-dimensional coordinates of the corrected key points. The first joint includes joints other than the metacarpophalangeal joints of the index, middle, ring and little fingers, and the second joint includes the metacarpophalangeal joints of the index, middle, ring and little fingers. The workstation is also used to calculate the first joint angle of each finger of the test hand based on the three-dimensional coordinates of the corrected key points using a vector inner product algorithm. The workstation is also used to determine a reference joint corresponding to the second joint in the first joint, and to obtain the three-dimensional coordinates of the key points after the reference joint is corrected, wherein the reference joint is a joint on the same finger as the second joint; Based on the normal vector of the palm plane and the three-dimensional coordinates of the key points after the reference joint correction, the three-dimensional coordinates of the projection point of the three-dimensional coordinates of the key points after the reference joint correction on the palm plane are obtained. The second joint angle is obtained by calculating the line connecting the three-dimensional coordinates of the projection point and the three-dimensional coordinates of the key point after the second joint correction, and the line connecting the three-dimensional coordinates of the key point after the reference joint correction and the three-dimensional coordinates of the key point after the second joint correction.

2. The intelligent hand function assessment system according to claim 1, characterized in that, The workstation is also used to determine whether the test subject's finger joints are hyperextended using the vector outer product algorithm and the right-hand screw rule. If hyperextension is present, the angle of hyperextension is recorded as a negative value corresponding to the angle of hyperextension.

3. The intelligent hand function assessment system according to claim 2, characterized in that, The workstation is also used to calculate the distance between thumbs facing each other and the distance between thumbs adducting.

4. The intelligent hand function assessment system according to claim 3, characterized in that, The workstation is also used to obtain the functional loss degree of each joint based on the calculated first joint angle, second joint angle, thumb opposition distance, thumb adduction distance, and joint hyperextension angle of each finger; calculate the functional loss degree of each finger based on the functional loss degree of each joint; calculate the overall functional loss degree of the hand based on the functional loss degree of each finger; and output the overall functional loss degree of the hand as the hand function assessment result.

5. The intelligent hand function assessment system according to claim 4, characterized in that, The workstation is also used to output the hand joint angle parameters of the test subject, as well as the hand image corresponding to the test gesture in the hand motion image and the coordinates of multiple initial key points in the hand image.

6. A method for intelligent hand function assessment based on the intelligent hand function assessment system according to any one of claims 1 to 5, characterized in that, The intelligent assessment method for hand function includes: The test demonstration video is obtained from the cloud server, and the display is controlled to play the test demonstration video for the tester, so as to guide the tester to demonstrate the joint movement of the fingers of the test hand according to the test gestures in the test demonstration video, wherein the cloud server obtains the test demonstration video based on the tester's personal information through a large language model; The optical sensor acquires real-time images of the test subject's hand movements during the test, showing the activity states of each joint, from multiple perspectives. The hand joint angle parameters of the test subject are obtained based on the hand motion images, and the hand function evaluation results of the test subject in this test are obtained and output based on the hand joint angle parameters; The step of obtaining the hand joint angle parameters of the test subject based on the hand motion image includes: Using a preset pose estimation algorithm, multiple initial key point coordinates are extracted from multiple hand images in the main view image of the hand motion image, and the multiple initial key point coordinates are corrected by the non-main view image of the hand motion image to obtain the corrected three-dimensional coordinates of the key points. Each hand includes 21 key points, which include 20 key points of each finger from the thumb to the little finger and 1 key point of the wrist. The 21 key points do not include the key points at the base of the second to fifth metacarpal bones. The coordinates of each initial key point include normalized coordinates and the distance of each key point relative to the wrist. The workstation is also used to calculate the first joint angle and the second joint angle of the test hand in the hand image based on the three-dimensional coordinates of the corrected key points. The first joint includes joints other than the metacarpophalangeal joints of the index, middle, ring and little fingers, and the second joint includes the metacarpophalangeal joints of the index, middle, ring and little fingers. The workstation is also used to calculate the first joint angle of each finger of the test hand based on the three-dimensional coordinates of the corrected key points using a vector inner product algorithm. The workstation is also used to determine a reference joint corresponding to the second joint in the first joint, and to obtain the three-dimensional coordinates of the key points after the reference joint is corrected, wherein the reference joint is a joint on the same finger as the second joint; Based on the normal vector of the palm plane and the three-dimensional coordinates of the key points after the reference joint correction, the three-dimensional coordinates of the projection point of the three-dimensional coordinates of the key points after the reference joint correction on the palm plane are obtained. The second joint angle is obtained by calculating the line connecting the three-dimensional coordinates of the projection point and the three-dimensional coordinates of the key point after the second joint correction, and the line connecting the three-dimensional coordinates of the key point after the reference joint correction and the three-dimensional coordinates of the key point after the second joint correction.

Images