Gesture recognition system and gesture recognition method

Abstract

The invention relates to the technical field of image capturing and processing, in particular to a gesture recognition system and a gesture recognition method.The gesture recognition system comprises a rehabilitation evaluation instrument and a camera which is provided with a display screen and is arranged in an image capturing window in five directions; the lower computer comprises an image processor and is used for image acquisition control, image preprocessing and data transmission; according to the scheme, high-precision gesture recognition and rehabilitation condition judgment are achieved through cooperative work of the rehabilitation evaluator, the lower computer and the upper computer; the rehabilitation evaluation instrument multidirectional camera collects data, and the data is preprocessed by the lower computer and then transmitted to the upper computer; the upper computer gesture recognition module accurately positions joint points, tracks tracks and judges bending changes, the action evaluation module compares actual data with preset data to evaluate actions, the motion function deep analysis module evaluates the flexibility, coordination and muscle strength of the joints, comprehensive and accurate data is provided for hand rehabilitation evaluation, and doctors are assisted in making scientific rehabilitation schemes.

Description

Technical Field

[0001] The present invention relates to the technical field of image capture and processing, and particularly relates to a gesture recognition system and a gesture recognition method. Background Art

[0002] A gesture recognition system is a system applied in the technical field of image capture and processing that can recognize and analyze hand movements. In the field of hand rehabilitation, it can provide important basis for the rehabilitation evaluation and treatment of patients, helping doctors accurately understand the recovery of patients' hand functions, so as to formulate more targeted rehabilitation programs;

[0003] However, in the data acquisition link of traditional analysis systems, only relying on single cameras or low-resolution cameras, it is difficult to comprehensively and clearly capture the subtle features of hand joints and the complete movement process, resulting in inaccurate subsequent analysis data. Moreover, traditional systems lack refined evaluation of movements and in-depth functional analysis, and it is difficult to comprehensively analyze various functions such as joint movement flexibility, coordination, and muscle strength, making the rehabilitation evaluation not comprehensive and in-depth enough, affecting the accuracy and effectiveness of rehabilitation treatment;

[0004] Therefore, the present invention proposes a gesture recognition system and a gesture recognition method. Through the coordinated operation of a rehabilitation evaluation instrument, a lower computer, and an upper computer, data is collected by multi-directional cameras, and high-precision gesture recognition and comprehensive rehabilitation status determination are achieved by combining advanced image processing algorithms and multi-module analysis functions. It can not only accurately locate joint points, track movement trajectories, and judge bending changes, but also quantitatively evaluate movements and conduct in-depth functional analysis, providing a more scientific and accurate basis for hand rehabilitation. Summary of the Invention

[0005] Technical problems to be solved: The problems of incomplete data acquisition, lack of refined evaluation of movements, and in-depth functional analysis.

[0006] In view of the deficiencies of the prior art, the present invention provides a gesture recognition system and a gesture recognition method, thereby solving the technical problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention is realized through the following technical solutions:

[0008] A gesture recognition system, comprising:

[0009] A rehabilitation evaluation instrument, provided with a display screen and cameras arranged in five directions within an image acquisition window;

[0010] A lower computer, including an image processor for image acquisition control, image preprocessing, and data transmission;

[0011] An upper computer, including a gesture recognition module, an action evaluation module, and a deep motion function analysis module;

[0012] The gesture recognition module includes a joint point extraction unit, a motion trajectory tracking unit, and a bending change judgment unit; the joint point extraction unit locates 21 hand joint points through convolution operations and activation functions; the motion trajectory tracking unit uses the state transition equation and the observation equation in combination with the Kalman filtering algorithm to track the motion trajectory of the joint points; the bending change judgment unit judges the bending state of the finger by calculating the change in the Euclidean distance between adjacent joint points of the same finger;

[0013] The action evaluation module includes an action setting and threshold determination unit, and an actual motion analysis and comparison unit; the action setting and threshold determination unit determines the motion trajectory thresholds of each action of the bone joints by collecting a large amount of action data of healthy people and patients at different rehabilitation stages; the actual motion analysis and comparison unit compares the actual motion data of the patient with the preset threshold to evaluate the quality of the action completion;

[0014] The in-depth motion function analysis module includes a joint motion flexibility analysis unit, a motion coordination analysis unit, and an indirect muscle strength evaluation unit; the joint motion flexibility analysis unit calculates the joint motion angle range based on the joint point motion trajectory to evaluate joint flexibility; the motion coordination analysis unit analyzes the motion synchronization and speed consistency between different joint points of the same finger and between different fingers to evaluate coordination; the indirect muscle strength evaluation unit indirectly evaluates muscle strength by calculating the joint point motion speed and acceleration.

[0015] In a possible implementation manner, the joint point extraction unit uses the convolution operation formula F l =σ(W l *F l-1 +b l ) and the activation function σ(x)=max(0,x) to accurately extract the hand joint point features from the input image and achieve precise positioning of 21 hand joint points;

[0016] The motion trajectory tracking unit uses the state transition equation X k =AX k-1 +Bu k +W k and the observation equation Z k =HX k +V k , in combination with the Kalman filtering algorithm, to accurately track the motion trajectory of the joint points and accurately reflect the continuous change of the joint point positions during the finger movement process;

[0017] The bending change judgment unit judges the bending or stretching state of the finger accurately by calculating the change in the Euclidean distance between adjacent joint points of the same finger and setting a reasonable threshold.

[0018] In a possible implementation, the action setting and threshold determination unit of the action evaluation module calculates the length of the motion trajectory by calculating the coordinate sequences of the fingertip and joint points in the x and y axis directions, and determines the motion trajectory threshold range;

[0019] The actual motion analysis and comparison unit compares the distance at the moment of fingertip contact and the length of the finger motion trajectory with the preset thresholds respectively through the joint point motion trajectory data, accurately evaluates the quality of action completion, and accurately determines whether there are rehabilitation problems.

[0020] In a possible implementation, the joint motion flexibility analysis unit calculates the angle change formed by adjacent joints through vector operations based on the joint point motion trajectory tracked by the gesture recognition module, and compares it with the joint angle change range of the normal population to determine whether the joint motion flexibility of the patient is limited;

[0021] The motion coordination analysis unit comprehensively evaluates the motion coordination by analyzing the time differences between different joint points of the same finger and between different fingers at the start, end, and intermediate key time points of the action, and simultaneously analyzing the consistency of the motion speeds of each joint point;

[0022] The muscle strength indirect evaluation unit calculates the motion speed and acceleration of the joint points during actions such as making a fist, compares the corresponding values with those of the normal population, indirectly evaluates the muscle strength, and judges the completion quality and efficiency of actions such as making a fist.

[0023] In a possible implementation, a gesture recognition method, the gesture recognition method includes the following steps:

[0024] Step 1, equipment preparation, ensure that all devices of the system are running normally;

[0025] Step 2, action execution, the user puts the hand into the imaging window according to the display screen prompt and completes multiple hand actions in sequence according to the instructions;

[0026] Step 3, action analysis, the camera captures the action image data, which is processed by the lower computer and then transmitted to the upper computer, and each module of the upper computer performs analysis;

[0027] Step 4, result display, after the upper computer analyzes, it gives the analysis results presented in the form of charts and text through the display screen.

[0028] In a possible implementation, the step 3 includes the following sub-steps:

[0029] Step 3.1, data reception and preprocessing: The lower computer transmits the image data after acquisition control and preprocessing to the upper computer. After the upper computer receives these data, it initializes the gesture recognition module to ensure that each unit is ready and can accurately process the hand action image data;

[0030] Step 3.2, Gesture Recognition Analysis: The joint point extraction unit of the gesture recognition module locates the hand joint points in the image based on convolutional operations and activation functions to determine the positions of 21 joint points; the motion trajectory tracking unit tracks the motion trajectories of the joint points during the entire action according to the state transition equation and the observation equation, in combination with the Kalman filtering algorithm; the bending change judgment unit calculates the change in the Euclidean distance between adjacent joint points of the same finger to determine the state changes such as finger bending and extension, and obtains complete gesture action information;

[0031] Step 3.3, Action Evaluation and Processing: The action setting and threshold determination unit of the action evaluation module calls the preset thresholds stored for different actions, and the actual motion analysis and comparison unit compares the joint point motion trajectory data obtained by the gesture recognition module with the preset thresholds of the corresponding actions; by calculating the distance at the moment of fingertip contact when the fingers are opposed in the action and comparing it with the preset contact distance threshold, and at the same time comparing the length of the finger motion trajectory with the preset threshold range, the quality of action completion is evaluated;

[0032] Step 3.4, In-depth Analysis of Motor Function: The joint motion flexibility analysis unit of the in-depth motor function analysis module calculates the range of motion angles of each joint in different actions based on the joint point motion trajectories tracked by the gesture recognition module to determine whether the joint motion flexibility is limited; the motion coordination analysis unit analyzes the motion synchronization between different joint points of the same finger and between different fingers, including the time differences at the start, end, and key intermediate time points of the action, and at the same time analyzes the consistency of the motion speeds of each joint point; the muscle strength indirect evaluation unit indirectly evaluates the muscle strength by calculating the motion speed and acceleration of the joint points in the action.

[0033] Beneficial Effects Compared with the Prior Art:

[0034] 1. In this solution, through the collaborative work of the rehabilitation evaluation instrument, the lower computer, and the upper computer, high-precision gesture recognition and rehabilitation status determination are achieved; the multi-directional cameras of the rehabilitation evaluation instrument collect data, which is preprocessed by the lower computer and then transmitted to the upper computer; the gesture recognition module of the upper computer accurately locates the joint points, tracks the trajectories, and judges the bending changes, the action evaluation module compares the actual and preset data to evaluate the action, and the in-depth motor function analysis module evaluates the joint flexibility, coordination, and muscle strength, providing comprehensive and accurate data for hand rehabilitation evaluation to assist doctors in formulating scientific rehabilitation plans;

[0035] 2. In this solution, the action threshold is set through the collection and analysis of a large number of sample data to achieve the quantitative evaluation of the patient's rehabilitation actions; in the action setting and threshold determination unit, the data of healthy people and patients are collected for different actions to determine the threshold; the actual motion analysis and comparison unit compares the patient's action data with the threshold to judge the quality of the action completion. For example, for the finger opposition action, it can accurately judge whether it meets the standard, timely discover rehabilitation problems, and make the rehabilitation evaluation more scientific and objective;

[0036] 3. In this solution, the comprehensive analysis of the patient's hand motor function is realized through the functions of multiple system modules; the gesture recognition module obtains the joint point information, and each unit of the in-depth motion function analysis module analyzes from aspects such as the flexibility, coordination of joint motion, and indirect evaluation of muscle strength; for example, analyzing the angle range of finger flexion and extension actions to evaluate flexibility, analyzing the motion synchronization of joint points to evaluate coordination, and calculating speed and acceleration to evaluate muscle strength, comprehensively understanding the patient's hand motor function, and providing a multi-dimensional reference basis for rehabilitation treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above description is only an overview of the technical solution of the present invention. In order to understand the technical means of the present invention more clearly and implement it according to the content of the description, the following takes the preferred embodiments of the present invention and combines the drawings to describe in detail as follows.

[0038] Figure 1 It is a schematic diagram of a rehabilitation evaluation instrument;

[0039] Figure 2 It is a framework diagram of a gesture recognition system;

[0040] Figure 3 It is a flowchart of the steps of a gesture recognition method. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] The preferred embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention can be implemented in various different forms, so the present invention is not limited to the embodiments described below;

[0042] The technical solutions in the embodiments of the present application are to solve the problems in the above background technology, and the general idea is as follows:

[0043] Embodiment 1:

[0044] Please refer to Figure 1 、 Figure 2As shown, this embodiment introduces a gesture recognition system, which is specially designed for hand rehabilitation assessment. Relying on the coordinated operation of the upper computer and the lower computer, high-precision gesture recognition and rehabilitation status determination are achieved. The core of the system hardware is a rehabilitation assessment instrument, which uses cameras arranged in multiple directions to collect hand movement data. The software level relies on the upper and lower computers to undertake corresponding calculation and processing tasks, realizing the full process function from data collection to evaluation result output;

[0045] 1. Rehabilitation assessment instrument (such as Figure 1 )

[0046] 1.1. Display screen, which is used to intuitively present various information fed back by the host computer. When the user performs rehabilitation assessment operations, the display screen will display detailed operation instructions, such as informing the user of the specific actions to be performed in the form of pictures and texts, such as "Please face up and slowly perform the five-finger grasping action";

[0047] After the assessment is completed, the scores of various assessment indicators can be displayed in a clear and easy-to-understand chart, which is convenient for doctors and patients to intuitively understand the progress of rehabilitation;

[0048] 1.2. The cameras are arranged in the imaging window in five directions: up, down, left, right, and front. With high resolution, they can clearly capture the subtle features of the hand joints, laying the foundation for accurate identification of joint points; for example, they can clearly distinguish the bending and wrinkling changes of the finger joints;

[0049] 60fps frame rate, 120° horizontal viewing angle, and 90° vertical viewing angle ensure accurate data collection and complete imaging area;

[0050] 1.3. Image taking window: There are two windows, which can be put in with both hands at the same time or with one hand;

[0051] 1.4. Dust cover: The dust cover is installed in front of the imaging window and is usually in a closed state. It can effectively prevent dust and impurities from entering the imaging window and affecting the imaging quality of the camera.

[0052] 2. Lower computer

[0053] 2.1. Hardware: NVIDIA Jetson Nano embedded processor, which has powerful computing power and rich interface resources, with quad-core Cortex-A57 CPU and 128-core Maxwell GPU, can quickly process large amounts of image data collected by the camera;

[0054] 2.2. Functional responsibilities:

[0055] 2.2.1. Image acquisition control: Connect to multiple cameras and synchronously control their startup, stop, and parameter adjustment; for example, automatically adjust parameters such as the exposure time and gain of the cameras according to the ambient light change to ensure stable quality of the acquired images.

[0056] 2.2.2. Image preprocessing: Perform preliminary processing on the original images acquired by the cameras, including operations such as denoising, grayscale conversion, and contrast enhancement; taking denoising as an example, use the Gaussian filtering algorithm to remove the random noise in the images, making the images clearer and facilitating subsequent joint point extraction.

[0057] 2.2.3. Data transmission: Transmit the preprocessed image data to the host computer, and at the same time receive the control instructions sent by the host computer, such as start / stop acquisition, adjust camera parameters, etc.

[0058] 3. Host computer, which includes a gesture recognition module, an action evaluation module, and a motion function in-depth analysis module.

[0059] 3.1. The gesture recognition module includes a joint point extraction unit, a motion trajectory tracking unit, and a bending change judgment unit.

[0060] 3.1.1. Joint point extraction unit

[0061] Let the input image be I, with its size being H×W×C (H is the height, W is the width, and C is the number of channels. For color images, C = 3); the output feature map F after passing through the l-th convolutional layer l can be obtained through the following convolutional operation: F l = σ(W l *F l-1 + b l ), where W l is the convolutional kernel weight matrix of the l-th layer. The size of the convolutional kernel is usually 3×3 or 5×5, etc. * represents the convolutional operation, which slides on the image and performs weighted summation of the convolutional kernel and the corresponding area. b l is the bias vector, adding a constant offset to each convolutional output result;

[0062] σ is the activation function, σ(x) = max(0, x), which can effectively solve the gradient vanishing problem and make the network have stronger learning ability;

[0063] For example, when the hand makes a fist gesture, after the input image passes through multiple convolutional layers, the model can extract features such as the edges and textures at the bending points of the finger joints. By continuously learning the relationship between these features and the joint point positions, the positions of 21 joint points of the hand can be accurately located, such as the pixel coordinates (x thumb-tip , y thumb-tip ) of the fingertip joint of the thumb in the image;

[0064] 3.1.2. Motion Trajectory Tracking Unit

[0065] Assume that the state vector X of the joint point at time k k includes the position p k and the velocity v k , that is

[0066] The state transition equation is: X k = AX k-1 + Bu k + W k ; where A is the state transition matrix. For two-dimensional planar motion, Δt is the time interval. In this embodiment, the camera frame rate is 60fps, B is the control input matrix, which can be set to 0 when there is no external control input; W k is the process noise, which follows a Gaussian distribution W k ~ N(0, Q), and Q is the process noise covariance matrix;

[0067] The observation equation is: Z k = HX k + V k , where Z k is the observed value, which is the joint point position detected in the image by the CNN algorithm in this embodiment; H is the observation matrix. For the case of only observing the position, H = [1 0]; V k is the observation noise, which follows a Gaussian distribution V k ~ N(0, R), and R is the observation noise covariance matrix;

[0068] For example, when the finger makes an opening and closing movement, the joint point position at the previous moment is used to predict the current moment position through the state transition equation. At the same time, combined with the actual joint point position (observed value) detected by the CNN in the current frame, the Kalman filtering algorithm is used to fuse the predicted value and the observed value to obtain a more accurate joint point position estimate, so as to smoothly track the joint point motion trajectory and accurately reflect the continuous change of the joint point position during the finger opening and closing process.

[0069] 3.1.3. Bending Change Judgment Unit

[0070] The bending situation is judged by calculating the change in the Euclidean distance between adjacent joint points on the same finger. Let the coordinates of adjacent joint points i and j on the same finger be (x i , y i ) and (x j , y j ), then the Euclidean distance d between them is: When the finger bends, the adjacent joint points get closer and the distance d decreases; when it extends, the distance increases;

[0071] For example, in the finger bending motion, for the adjacent joint point between the proximal phalanx and the middle phalanx, if the distance at the initial state is After the finger bends to a certain extent, the distance becomes If d1 < d0, it can be determined that the finger joint has a bending motion. By setting a reasonable distance change threshold, such as Δd = 0.8d0 (this threshold is obtained through statistical analysis of a large number of sample data), when d1 < Δd, it can be accurately determined that the finger is in a bent state;

[0072] 3.2. The motion evaluation module includes a motion setting and threshold determination unit, and an actual motion analysis and comparison unit;

[0073] 3.2.1. Motion setting and threshold determination unit

[0074] For the five motions preset in the system, through the acquisition and analysis of the motion data of a large number of healthy people and patients at different rehabilitation stages, determine the motion trajectory thresholds between the bone joints of each motion; taking the finger opening and closing motion as an example, when collecting data, record the motion trajectory of the fingertip joint point P of the index finger during the process from fully spreading the five fingers to fully closing them index-tip Its coordinate sequence in the x-axis direction is Its coordinate sequence in the y-axis direction is Calculate its motion trajectory length Perform statistical analysis on the L values calculated for a large number of samples to obtain the mean μ and the standard deviation σ, and set the threshold range as [μ - ασ, μ + ασ], where α is a coefficient determined based on experience and experiments, such as α = 1.5. This threshold range will be used as a reference standard for subsequent evaluation of the rehabilitation of the patient's finger opening and closing motion;

[0075] 3.2.2. Actual motion analysis and comparison unit

[0076] When the patient performs the rehabilitation motion, the host computer obtains the joint point motion trajectory data in real time and compares it with the preset threshold; taking the evaluation of the finger opposition motion as an example, let the fingertip joint point of the little finger be P little-tip , and the fingertip joint point of the thumb be P thumb-tip . When performing the finger opposition motion, calculate the distance between them at the moment of contact

[0077] At the same time, record the motion trajectory length L of the joint point of the little finger from the initial position to the contact with the thumb little ; Compare d contact with the preset contact distance threshold d threshold , and compare L little with the preset motion trajectory length threshold range [L min , L maxComparison;

[0078] If |d contact - d threshold | < ∈1 and L min ≤ L little ≤ L max (∈1 is the allowable small distance deviation), it is determined that the finger opposition movement is completed with good quality; if the above conditions are not met, it is prompted that there may be rehabilitation problems, such as abnormal movement trajectory length caused by limited joint movement, or excessive contact distance deviation caused by insufficient muscle strength, etc.;

[0079] 3.3. The motion function in-depth analysis module includes a joint motion flexibility analysis unit, a motion coordination analysis unit, and an indirect muscle strength assessment unit;

[0080] 3.3.1. Joint motion flexibility analysis unit

[0081] Based on the joint point motion trajectories tracked by the gesture recognition module, calculate the motion angle ranges of each joint in different actions; taking the finger joint as an example, let the joint points between the proximal phalanx and the middle phalanx of a finger be A and B; when performing finger flexion and extension actions, as time goes by, the joint point positions change continuously; for example, at the starting moment of a finger extension action, the coordinates of point A are (x1, y1), and the coordinates of point B are (x2, y2); in the middle of the action, the coordinates of point A become (x3, y3), and the coordinates of point B become (x4, y4);

[0082] Calculate the angle change formed by adjacent joints through vector operations. For example, the vector Use the vector dot product formula Calculate the change in angle θ;

[0083] Assume that in the finger extension action of the normal population, the angle change range of this joint is usually between 0° and 90°; if the patient calculates an angle change range of only 0° to 60° when performing this action, it indicates that the joint motion flexibility is limited, and there may be problems such as joint stiffness, affecting the normal extension degree of the finger;

[0084] 3.3.2. Motion coordination analysis unit

[0085] Analyze the motion synchronization between different joint points on the same finger and between different fingers. For multiple joint points on the same finger, calculate the time differences at the start, end, and intermediate key time points of the action;

[0086] For example, during a finger bending motion, record the times t1, t2, and t3 when the joint points of the proximal phalanx, middle phalanx, and distal phalanx start to bend; normally, the time differences between the start of bending of these three joint points should be within a very short time range, assumed to be within 0.1 seconds; if the calculated Δt 12 = t2 - t1 = 0.3 seconds, and Δt 23 = t3 - t2 = 0.2 seconds, which far exceeds the normal range. This indicates that the movement coordination of each joint of this finger is poor, and there may be problems in muscle control or nerve conduction, resulting in the joints being unable to complete the bending motion cooperatively;

[0087] For different fingers, also calculate the time differences in the movement of each finger joint point when performing the same action (such as finger opening and closing); for example, during the finger opening and closing action, the time when the joint point of the index finger starts to open is t4, and the time when the joint point of the middle finger starts to open is t5. The normal time difference may be within 0.05 seconds; if the patient's t5 - t4 = 0.2 seconds, it indicates that the coordination between the index finger and the middle finger in this action is poor, affecting the smoothness of the entire hand movement;

[0088] In addition, the consistency of the movement speeds of each joint point can also be analyzed to further evaluate the movement coordination; for example, during the finger opening and closing process, normally the movement speeds of each finger joint point should be similar; if the movement speed of the joint point of the ring finger of a certain patient is significantly faster than that of other fingers, it will cause the finger opening and closing action to be asymmetric, affecting the normal functioning of the hand;

[0089] 3.3.3. Indirect Muscle Strength Assessment Unit

[0090] In some actions, such as the fist clenching action, the magnitude of muscle strength will affect the movement speed and acceleration of the joint point; assume that the movement trajectory of the joint point within a certain period of time is {(x i , y i )} i = 1 n , and calculate the speed v i and acceleration a i of the joint point through numerical differentiation; for example, the speed v i can be approximately calculated by dividing the difference in the joint point positions at adjacent times by the time interval: The acceleration a i is then calculated through the rate of change of speed;

[0091] Assume that during the fist clenching action, the average speed of the joint point of the normal population can reach a certain value. For example, if the fist clenching action is completed within 0.5 seconds, the average speed of the joint point is v0; if when the patient completes the fist clenching action within the same time, the calculated average speed of the joint point is only 0.5v0, it indicates that their muscle strength is insufficient and they cannot drive the joint movement as quickly and powerfully as a normal person;

[0092] Similarly, when analyzing acceleration, normal people may have a relatively large acceleration at the initial stage of fist clenching, quickly bending their fingers. If the joint point acceleration of a patient is significantly lower than the normal range, it also reflects that their muscle strength is weak and it is difficult to generate enough power to quickly change the joint movement state, affecting the completion quality and efficiency of the fist clenching action.

[0093] Embodiment 2:

[0094] Please refer to Figure 3 As shown, this embodiment is the gesture recognition method of the gesture recognition system in Embodiment 1. The method specifically includes the following steps:

[0095] Step 1: Equipment preparation

[0096] First, ensure that all devices of the gesture recognition system are running normally;

[0097] Step 2: Action execution. The user places their hand in the imaging window according to the prompts on the display screen. The display screen will give detailed instructions in a vivid and graphic form, guiding the user to complete five actions in sequence according to the instructions.

[0098] Step 2.1: Finger bending and stretching; palm facing up, slowly grasping and slowly opening the five fingers: The user places their hand steadily in the imaging window with the palm facing up. According to the prompts on the display screen, first slowly perform the five-finger stretching action, feeling the process of the fingers from being completely curled to straightening, and then slowly perform the five-finger grasping action to gradually bend the fingers into a fist. Each action should be as standard and steady as possible, and repeat it multiple times.

[0099] Step 2.2: Finger opening and closing; palm facing up, opening the five fingers and slowly closing them towards the middle, then opening them again: Keep the palm facing up. Under the indication of the display screen, open the five fingers to the maximum extent, then slowly close them towards the middle to make the five fingers close, and then slowly open them again. The whole process should be carried out at a uniform speed, and pay attention to the coordination of finger movements.

[0100] Step 2.3: Finger opposing; palm facing up, by bending the fingers, sequentially touching the fingertips of the little finger, ring finger, middle finger, and index finger with the fingertip of the thumb: According to the prompts on the display screen, place the hand with the palm facing up. First bend the little finger to gently touch the fingertip of the thumb, and then sequentially complete the opposing actions of the ring finger, middle finger, index finger, and thumb with the thumb. Ensure that the fingertips accurately touch in each opposing action, and keep the palm stable during the action process.

[0101] Step 2.4, Adduction and abduction of the thumb; with the palm facing up, slowly bring the thumb closer to and away from the index finger: Under the instruction of the display screen, with the palm facing up, slowly bring the thumb closer to the index finger and feel the adduction movement of the thumb, then slowly open it and return to the initial position, repeating the operation while paying attention to the smoothness and accuracy of the thumb movement.

[0102] Step 2.5, Flexion and extension of the thumb; with the palm facing up, slowly bring the thumb closer to the palm and then slowly move it away: According to the prompt of the display screen, with the palm facing up, slowly bend the thumb closer to the palm and then slowly extend it away, returning to the initial state, repeating the operation and carefully experiencing the movement process of the thumb.

[0103] Step 3, Motion analysis. After the motion is completed, the camera transmits the image data of the entire process of the hand motion it has captured to the lower computer processor; the processor processes this image data and then transmits it to the upper computer;

[0104] Step 3.1, After the upper computer receives the data, the gesture recognition module starts to work; the joint point extraction unit uses convolution operations and activation functions to accurately extract the position information of 21 hand joint points from the image; the motion trajectory tracking unit uses the state transition equation and the observation equation, combined with the Kalman filter algorithm, to smoothly track the motion trajectories of the joint points; the bending change judgment unit calculates the change in the Euclidean distance between adjacent joint points of the same finger to determine the bending or stretching state of the finger.

[0105] Step 3.2, The motion evaluation module compares the actual motion trajectory with the preset thresholds for different motions based on the data obtained by the gesture recognition module; for example, in the finger opposition motion, calculate the distance at the moment when the fingertips touch during finger opposition and compare it with the preset contact distance threshold, and at the same time compare the length of the finger motion trajectory with the preset threshold range to judge the quality of the motion completion.

[0106] Step 3.3, The motion function in-depth analysis module deeply analyzes the hand motion function from multiple dimensions based on the results of the gesture recognition module; the joint motion flexibility analysis unit calculates the motion angle range of each joint in different motions to judge whether the joint motion flexibility is limited; the motion coordination analysis unit analyzes the motion synchronization and speed consistency between different joint points of the same finger and between different fingers to evaluate the motion coordination; the muscle strength indirect evaluation unit indirectly evaluates the muscle strength by calculating the motion speed and acceleration of the joint points during the motion.

[0107] Step 4, Result display

[0108] After comprehensive analysis, the host computer gives detailed analysis results through the display screen of the rehabilitation evaluator; the results are presented in the form of clear and easy-to-understand charts, including the scores of various evaluation indicators, such as the score of finger joint flexibility, the score of movement coordination, the score of muscle strength, etc.; at the same time, a written description is made for the completion of each action, pointing out the problems existing in the action, such as abnormal movement trajectories caused by limited joint activities, excessive contact distance deviation caused by insufficient muscle strength, and poor joint movement coordination affecting the fluency of hand movements, etc.; these results provide intuitive information on the progress of hand rehabilitation for doctors and patients, helping doctors formulate more accurate rehabilitation treatment plans, and also facilitating patients to understand their own hand function status and actively cooperate with rehabilitation training.

[0109] Finally, it should be noted that: Obviously, the above embodiments are only examples for clearly explaining the present invention, rather than limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation manners here. And the obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A gesture recognition system, characterized in that, Including: A rehabilitation evaluation instrument, which is provided with a display screen and cameras arranged in five directions within an imaging window; A lower computer, including an image processor for image acquisition control, image preprocessing, and data transmission; An upper computer, including a gesture recognition module, an action evaluation module, and a motion function in-depth analysis module; The gesture recognition module includes a joint point extraction unit, a motion trajectory tracking unit, and a bending change judgment unit; the joint point extraction unit locates 21 hand joint points through convolution operations and activation functions; the motion trajectory tracking unit uses a state transition equation and an observation equation in combination with a Kalman filtering algorithm to track the motion trajectory of joint points; the bending change judgment unit judges the finger bending state by calculating the change in the Euclidean distance between adjacent joint points of the same finger; The action evaluation module includes an action setting and threshold determination unit, and an actual motion analysis and comparison unit; the action setting and threshold determination unit determines the motion trajectory thresholds of each action bone joint by collecting the action data of a large number of healthy people and patients at different rehabilitation stages; the actual motion analysis and comparison unit compares the actual motion data of the patient with the preset threshold to evaluate the quality of action completion; The motion function in-depth analysis module includes a joint motion flexibility analysis unit, a motion coordination analysis unit, and a muscle strength indirect evaluation unit; The joint motion flexibility analysis unit calculates the joint motion angle range based on the joint point motion trajectory to evaluate joint flexibility; the motion coordination analysis unit analyzes the motion synchronization and speed consistency between different joint points of the same finger and between different fingers to evaluate coordination; the muscle strength indirect evaluation unit indirectly evaluates muscle strength by calculating the joint point motion speed and acceleration.

2. The gesture recognition system according to claim 1, wherein: The joint point extraction unit extracts the hand joint point features accurately from the input image through the convolution operation formula F l =σ(W l *F l-1 +b l ) and the activation function σ(x)=max(0,x), and realizes the precise positioning of 21 hand joint points; The motion trajectory tracking unit uses the state transition equation X k = AX k-1 + Bu k + W k and the observation equation Z k = HX k + V k , combines with the Kalman filtering algorithm to accurately track the motion trajectory of the joint points and precisely reflect the continuous change of the joint point positions during the finger movement process; The bending change judgment unit accurately judges the bending or stretching state of the finger by calculating the change of the Euclidean distance between adjacent joint points of the same finger and setting a reasonable threshold value.

3. The gesture recognition system according to claim 1, wherein: The action setting and threshold determination unit of the action evaluation module determines the motion trajectory threshold range by calculating the motion trajectory length of the fingertip and joint points in the x and y axis directions; The actual motion analysis and comparison unit compares the distance at the moment of fingertip contact and the finger motion trajectory length with the preset threshold respectively through the joint point motion trajectory data to accurately evaluate the quality of action completion and accurately judge whether there are rehabilitation problems.

4. The gesture recognition system according to claim 1, wherein: The joint motion flexibility analysis unit calculates the angle change formed by adjacent joints at adjacent moments through vector operations based on the joint point motion trajectory tracked by the gesture recognition module, and compares it with the joint angle change range of the normal population to judge whether the joint motion flexibility of the patient is limited; The motion coordination analysis unit comprehensively evaluates motion coordination by analyzing the time difference between different joint points of the same finger and between different fingers at the start, end, and intermediate key time points of the action, and at the same time analyzing the consistency of the motion speeds of each joint point; The muscle strength indirect evaluation unit indirectly evaluates muscle strength by calculating the joint point motion speed and acceleration in actions such as fist clenching, compares it with the corresponding values of the normal population, and judges the quality and efficiency of actions such as fist clenching.

5. A gesture recognition method applied to the gesture recognition system according to any one of claims 1-4, characterized in that, The described gesture recognition method includes the following steps: Step 1, device preparation, ensuring the normal operation of all devices in the system; Step 2, action execution, the user puts the hand into the imaging window according to the display prompt and sequentially completes multiple hand actions according to the instructions; Step 3, action analysis, the camera captures action image data, which is processed by the lower computer and then transmitted to the upper computer, and each module of the upper computer conducts analysis; Step 4, result display, after the upper computer analyzes, it gives the analysis results presented in the form of charts and text through the display.

6. The gesture recognition method according to claim 5, wherein The said Step 3 includes the following sub-steps: Step 3.1, data reception and preprocessing: The lower computer transmits the image data that has undergone acquisition control and preprocessing to the upper computer. After the upper computer receives these data, it initializes the gesture recognition module to ensure that each unit is ready and can accurately process the hand action image data; Step 3.2, gesture recognition analysis: The joint point extraction unit of the gesture recognition module locates the hand joint points in the image based on convolutional operations and activation functions to determine the positions of 21 joint points; the motion trajectory tracking unit tracks the motion trajectories of the joint points during the entire action process according to the state transition equation and the observation equation, combined with the Kalman filtering algorithm; the bending change judgment unit calculates the change in the Euclidean distance between adjacent joint points of the same finger to judge the state changes such as finger bending and stretching, and obtains the complete gesture action information; Step 3.3, action evaluation and processing: The action setting and threshold determination unit of the action evaluation module calls the preset thresholds stored for different actions. The actual motion analysis and comparison unit compares the joint point motion trajectory data obtained by the gesture recognition module with the preset thresholds of the corresponding actions; by calculating the distance at the moment of fingertip contact when the fingers are opposed in the action and comparing it with the preset contact distance threshold, and at the same time comparing the length of the finger motion trajectory with the preset threshold range, the quality of the action completion is evaluated; Step 3.4, in-depth analysis of motor function: The joint motion flexibility analysis unit of the in-depth motor function analysis module calculates the motion angle range of each joint in different actions based on the joint point motion trajectories tracked by the gesture recognition module to judge whether the joint motion flexibility is limited; the motion coordination analysis unit analyzes the motion synchronization between different joint points of the same finger and between different fingers, including the time differences at the start, end, and intermediate key time points of the action, and at the same time analyzes the consistency of the motion speeds of each joint point; The muscle strength indirect evaluation unit indirectly evaluates the muscle strength by calculating the motion speed and acceleration of the joint points in the action.

Images