A hand rehabilitation assessment method, system and device based on gesture score

Abstract

The present application relates to the field of image processing, and provide a kind of hand rehabilitation assessment method, system and equipment based on gesture score, including obtaining spatial input feature and time domain input feature, spatial graph convolution and time graph convolution are carried out to obtain spatial convolution output and time convolution output;Average pooling and full connection are carried out to obtain space-time attention output;Phase division is carried out to obtain stage output, multi-scale residual graph convolution and global average pooling are carried out to obtain identification image;Hand three-dimensional data is obtained, joint angle is calculated, thumb finger pair distance is calculated, and finger function score is calculated according to joint angle and thumb finger pair distance;Hand range of motion score and gesture speed score are calculated, gesture accuracy score is calculated, gesture score is calculated according to hand range of motion score, gesture accuracy score and gesture speed score;Weighted sum is obtained to obtain hand activity score, and the degree of hand rehabilitation is evaluated according to hand activity score.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a hand rehabilitation assessment method, system and device based on gesture scores. Background Technology

[0002] In gesture recognition, existing technologies employ Long Short-Term Memory (LSTM) networks. These networks effectively learn the dependencies between hand joints over time. However, when modeling long sequences, information forgetting occurs, and recognition accuracy is limited in complex environments. Dynamic gesture recognition can also be achieved using 3D convolutional neural networks. While this model can dynamically recognize spatiotemporal features, it demands significant computational and memory resources. Extended and supplemented displacement-multi-scale G3D neural networks can also be used for action recognition. These networks effectively capture subtle patterns of movement through multi-scale spatiotemporal convolution, but the algorithm cannot capture key features, limiting its efficiency in action recognition.

[0003] In the area of ​​hand rehabilitation assessment, many researchers have proposed numerous representative assessment methods. These include using exoskeletons to record the torque and angles of hand joints during flexion and extension movements to assess hand health; employing novel infrared imaging devices to measure the angles of multiple hand joints; and performing three specific movements consecutively to complete a hand health assessment. However, since normal daily functioning largely depends on the coordinated movement of various joints, and the aforementioned assessment methods only focus on individual joints without considering the overall coordination between joints, this limits the accuracy of the assessment results. Summary of the Invention

[0004] The present invention aims to at least solve one of the technical problems existing in related technologies. To this end, the present invention provides a hand rehabilitation assessment method, system, and device based on gesture scores, which analyzes hand movement images to obtain three-dimensional hand data, thereby calculating a hand activity score to evaluate the degree of hand rehabilitation.

[0005] This invention provides a hand rehabilitation assessment method based on gesture scores, comprising:

[0006] S1: Acquire hand motion images, extract spatial and temporal input features from the hand motion images, perform spatial graph convolution on the spatial input features to obtain spatial convolution output, and perform temporal graph convolution on the temporal input features to obtain temporal convolution output;

[0007] S2: Perform average pooling and full connection on the spatial convolution output and temporal convolution output to obtain the spatiotemporal attention output. Divide the spatiotemporal attention output into stages to obtain the stage output. Perform multi-scale residual map convolution and global average pooling on the stage output to obtain the recognition image.

[0008] S3: Obtain three-dimensional hand data by recognizing images, calculate joint angles and thumb-to-finger distance based on the three-dimensional hand data, and calculate finger function score based on joint angles and thumb-to-finger distance;

[0009] S4: Determine the demonstration gesture data, calculate the hand range of motion score and gesture speed score, calculate the gesture accuracy score based on the demonstration gesture data, and calculate the gesture score based on the hand range of motion score, gesture accuracy score, and gesture speed score;

[0010] S5: Weighted summation of finger function score and gesture score to obtain hand activity score, and assessment of hand rehabilitation level based on hand activity score.

[0011] According to the hand rehabilitation assessment method based on gesture scores provided by the present invention, in step S1, spatial features are extracted from the hand movement image to obtain the spatial input features, and temporal features are extracted from the hand movement image to obtain the temporal input features.

[0012] According to the hand rehabilitation assessment method based on gesture scores provided by the present invention, in step S1, a spatial adjacency matrix and a temporal adjacency matrix are determined, and a spatial graph convolution is performed on the spatial input features through the spatial adjacency matrix to obtain the spatial convolution output. The temporal input features are then convolved with the temporal graph through the temporal adjacency matrix to obtain the temporal convolution output.

[0013] According to the hand rehabilitation assessment method based on gesture score provided by the present invention, in step S2, the spatial convolution output is subjected to spatial average pooling to obtain spatial average pooling output, the temporal convolution output is subjected to temporal average pooling to obtain temporal average pooling output, the spatial average pooling output and the temporal average pooling output are fully connected to obtain fully connected output, the fully connected output is weighted outer product to obtain spatiotemporal attention matrix, and the spatiotemporal attention output is obtained through the spatiotemporal attention matrix.

[0014] According to the hand rehabilitation assessment method based on gesture score provided by the present invention, in step S2, the action stage is determined, the feature dimension is determined according to the action stage, the spatiotemporal attention output is divided into stages according to the feature dimension and the action stage to obtain the stage output, and the stage output is subjected to multi-scale residual map convolution and global average pooling to obtain the recognition image.

[0015] According to the hand rehabilitation assessment method based on gesture scores provided by the present invention, step S3 further includes:

[0016] S31: Obtain the three-dimensional data of the hand, including the hand joint coordinates, based on the recognized image, and calculate the hand bone length and bone vector based on the hand joint coordinates;

[0017] S32: Determine the theoretical maximum angle of the joint, calculate the joint angle based on the bone vector and the length of the hand bones, and obtain the joint flexion score based on the joint angle and the theoretical maximum angle of the joint;

[0018] S33: Determine the maximum distance between thumb and index finger, calculate the distance between thumb and index finger based on the hand joint coordinates, calculate the thumb alignment score based on the maximum distance between thumb and index finger and the distance between thumb and index finger, and obtain the finger function score through the thumb alignment score and the joint flexion score.

[0019] According to the hand rehabilitation assessment method based on gesture scores provided by the present invention, step S4 further includes:

[0020] S41: Determine the demonstration gesture data. Theoretical maximum range of motion and theoretical maximum joint velocity are determined. The joint displacement range is obtained using the three-dimensional hand data. A hand range of motion score is calculated based on the joint displacement range and the theoretical maximum range of motion.

[0021] S42: Calculate the joint movement speed, and obtain the gesture speed score based on the joint movement speed and the theoretical maximum joint movement speed;

[0022] S43: Obtain the standard joint position from the demonstration gesture data, calculate the motion accuracy based on the standard joint position, obtain the gesture accuracy score through the motion accuracy, and take the average score of the hand range of motion score, gesture accuracy score and gesture speed score as the gesture score.

[0023] According to the hand rehabilitation assessment method based on gesture scores provided by the present invention, in step S5, multiple typical gestures are determined, hand movement images when performing typical gestures are acquired, and steps S1 to S4 are executed to obtain multiple finger function scores and gesture scores. Ridge regression is performed on the multiple finger function scores and gesture scores to obtain a weight matrix. The hand activity score is obtained by weighted summation through the weight matrix.

[0024] The present invention also provides a hand rehabilitation assessment system based on gesture scoring, comprising:

[0025] Spatiotemporal convolution module: used to acquire hand action images, extract spatial input features and temporal input features from the hand action images, perform spatial graph convolution on the spatial input features to obtain spatial convolution output, and perform temporal graph convolution on the temporal input features to obtain temporal convolution output;

[0026] Image recognition module: It is used to perform average pooling and full connection on the spatial convolution output and temporal convolution output to obtain spatiotemporal attention output, divide the spatiotemporal attention output into stages to obtain stage output, and perform multi-scale residual map convolution and global average pooling on the stage output to obtain the recognition image.

[0027] Finger function scoring module: used to obtain three-dimensional hand data by recognizing images, calculate joint angles and thumb-to-finger distance based on the three-dimensional hand data, and calculate finger function score based on joint angles and thumb-to-finger distance;

[0028] Gesture scoring module: used to determine demonstration gesture data, calculate hand range of motion score and gesture speed score, calculate gesture accuracy score based on demonstration gesture data, and calculate gesture score based on hand range of motion score, gesture accuracy score and gesture speed score;

[0029] Hand activity scoring module: This module is used to weight and sum the finger function score and gesture score to obtain a hand activity score, which is then used to assess the degree of hand rehabilitation.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a hand rehabilitation assessment method based on gesture scores as described above.

[0031] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0032] This invention provides a hand rehabilitation assessment method, system, and device based on gesture scores. It performs spatial and temporal convolution on hand movement images to model their spatial and temporal correlations, respectively. Stage division captures the continuous variation patterns of hand movements, thus achieving multi-scale spatiotemporal features on continuous hand movements—that is, the efficient fusion of local details and global motion. Finger function scores and gesture scores provide a comprehensive assessment of local joints and overall coordination mechanisms. Furthermore, ridge regression is used to assign weights to each gesture, improving the accuracy of hand rehabilitation assessment and making hand activity scores more practical and medically relevant.

[0033] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a hand rehabilitation assessment method based on gesture scoring provided by the present invention.

[0036] Figure 2 This is a schematic diagram of the structure of a hand rehabilitation assessment system based on gesture scoring provided by the present invention.

[0037] Figure 3 This is a schematic diagram of the structure of a hand rehabilitation assessment device based on gesture scoring provided by the present invention.

[0038] Figure label:

[0039] 100. Spatiotemporal convolution module; 200. Image recognition module; 300. Finger function scoring module; 400. Gesture scoring module; 500. Hand activity scoring module; 810. Processor; 820. Communication interface; 830. Memory; 840. Communication bus. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0041] In the description of the embodiments of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0042] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0043] In embodiments of the present invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0044] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0045] The following is combined Figures 1 to 3 Specific embodiments of the present invention are described below. Figure 1A flowchart illustrating a hand rehabilitation assessment method based on gesture scores provided by this invention includes:

[0046] S1: Acquire hand motion images, extract spatial and temporal input features from the hand motion images, perform spatial graph convolution on the spatial input features to obtain spatial convolution output, and perform temporal graph convolution on the temporal input features to obtain temporal convolution output;

[0047] Furthermore, the objective of this stage is to obtain spatial input features and temporal input features, and then perform spatiotemporal convolution on them to obtain spatial convolution output and temporal convolution output. Specifically, in step S1, spatial features are extracted from the hand action image to obtain the spatial input features, and temporal features are extracted from the hand action image to obtain the temporal input features.

[0048] In step S1, the spatial adjacency matrix and the temporal adjacency matrix are determined. The spatial input features are convolved with the spatial graph using the spatial adjacency matrix to obtain the spatial convolution output. The temporal input features are convolved with the temporal graph using the temporal adjacency matrix to obtain the temporal convolution output.

[0049] The specific implementation method for the above steps in this embodiment is as follows:

[0050] First, it is necessary to acquire hand motion images. These hand motion images are images obtained by the subject performing a specified gesture, captured by cameras placed in multiple directions. To avoid incomplete information acquisition due to occlusion, cameras can be placed in front of, to the side of, and above the subject's hand to capture hand motion images.

[0051] Next, the spatial and temporal features included in the hand motion image need to be extracted separately. Spatial feature extraction is performed on the hand motion image to obtain the spatial input features. Temporal features are extracted from hand motion images to obtain temporal input features. Next, we need to determine the spatial adjacency matrix of order d. and the d-th order time adjacency matrix This allows us to perform spatial graph convolution and temporal graph convolution on the spatial input features and temporal input features respectively, thereby obtaining the spatial convolution outputs. Temporal convolution output :

[0052]

[0053] Where D represents the spatial extent of the neighborhood map of the hand action image. Let d be the space normalization matrix of order d. Let d be the time-normalized matrix of order d. Let d be the weight vector of the convolution operation. Let d be the mask matrix. This represents the dot product operation.

[0054] S2: Perform average pooling and full connection on the spatial convolution output and temporal convolution output to obtain the spatiotemporal attention output. Divide the spatiotemporal attention output into stages to obtain the stage output. Perform multi-scale residual map convolution and global average pooling on the stage output to obtain the recognition image.

[0055] Furthermore, the objective of this stage is to obtain the spatiotemporal attention output, divide it into stages to obtain stage outputs, and then perform multi-scale residual map convolution and global average pooling to obtain the recognized image. Specifically, in step S2, spatial average pooling is performed on the spatial convolution output to obtain a spatial average pooled output, temporal average pooling is performed on the temporal convolution output to obtain a temporal average pooled output, a fully connected layer is formed on the spatial average pooled output and the temporal average pooled output to obtain a fully connected output, and a weighted outer product is performed on the fully connected output to obtain a spatiotemporal attention matrix, through which the spatiotemporal attention output is obtained.

[0056] In step S2, the action stage is determined, the feature dimension is determined based on the action stage, the spatiotemporal attention output is divided into stages based on the feature dimension and the action stage to obtain the stage output, and the stage output is subjected to multi-scale residual map convolution and global average pooling to obtain the recognition image.

[0057] The specific implementation method for the above steps in this embodiment is as follows:

[0058] First, the spatial convolution output and the temporal convolution output need to be averaged to obtain the spatial averaged pooled output. and time-averaged pooling output :

[0059]

[0060] in, It is the space average pooling function. This is the time-average pooling function. Then, a fully connected function is applied to the spatial average pooling output and the time-average pooling output to obtain the fully connected output O:

[0061]

[0062] in, The first learnable bias term. The second learnable bias term. This indicates element-wise addition. The first learnable weight matrix, Let be the learnable second weight matrix, and sigmoid() be the sigmoid activation function.

[0063] Next, a weighted outer product is performed on the fully connected outputs to obtain the spatiotemporal attention matrix. :

[0064]

[0065] in, The third bias term is learnable. The fourth bias term is learnable. This represents the outer product operation. The third weight matrix is ​​learnable. This is a learnable fourth weight matrix. The HardSwish activation function is used. Finally, the spatiotemporal attention output is obtained. :

[0066]

[0067] in, The input features for hand motion images include both spatial and temporal input features. Here, 1×1, 5×1, and 7×1 convolutional kernels are used to process and extract input features from single frames, 5 frames, and 7 frames respectively. After each extraction, a 3×1 max pooling operation is performed on the extracted features to refine and compress them. Finally, the processed feature extraction results at different time scales are combined to obtain input features that can be used for recognition at different time scales and improve recognition accuracy, which are then used to obtain the spatiotemporal attention output.

[0068] Next, it is necessary to determine the action stages of the hand motion images. In this embodiment, it is divided into five stages: the initial stage, the acceleration stage, the peak stage, the stabilization stage, and the final stage. This divides a series of hand motion images into five stages. Each stage is assigned its own feature dimension based on the duration of each part. This allows for the segmentation of the spatiotemporal attention output based on the feature dimensions and action stages, ensuring that the feature dimension assigned to each stage is determined. Here, the feature dimension is... C represents the number of channels for spatiotemporal attention output, T represents the length of each stage, and K represents the number of hand joints. This allows us to obtain the stage output of the i-th stage. Subsequently, multi-scale residual map convolution is performed on the output of each stage to obtain the multi-scale residual map of the i-th stage. :

[0069]

[0070] in, To perform multi-scale residual map convolution, the specific process of multi-scale residual map convolution is to convolve the input using the formulas for spatial map convolution and temporal map convolution in step S1, and then fuse the outputs of the formulas for spatial map convolution and temporal map convolution to obtain the output multi-scale residual map.

[0071] Finally, global average pooling is used to obtain the recognized image. :

[0072]

[0073] in, This process involves first performing global average pooling on the content and then receiving 5-hop neighbor feature information before concatenating the elements. The resulting recognition image establishes connections from detailed to macroscopic levels, significantly expanding the equivalent receptive field of the terminal segment. Furthermore, it captures the spatiotemporal features between nodes at different distances, enabling the recognition image to simultaneously consider both short-term dynamics and long-term spatiotemporal dependencies between joints, thus improving the accuracy of the recognition image and accurately identifying each joint and determining its position.

[0074] S3: Obtain three-dimensional hand data by recognizing images, calculate joint angles and thumb-to-finger distance based on the three-dimensional hand data, and calculate finger function score based on joint angles and thumb-to-finger distance;

[0075] Furthermore, the objective of this stage is to calculate joint angles and thumb-to-finger distance based on three-dimensional hand data, thereby calculating a finger function score. Specifically, step S3 further includes:

[0076] S31: Obtain the three-dimensional data of the hand, including the hand joint coordinates, based on the recognized image, and calculate the hand bone length and bone vector based on the hand joint coordinates;

[0077] S32: Determine the theoretical maximum angle of the joint, calculate the joint angle based on the bone vector and the length of the hand bones, and obtain the joint flexion score based on the joint angle and the theoretical maximum angle of the joint;

[0078] S33: Determine the maximum distance between thumb and index finger, calculate the distance between thumb and index finger based on the hand joint coordinates, calculate the thumb alignment score based on the maximum distance between thumb and index finger and the distance between thumb and index finger, and obtain the finger function score through the thumb alignment score and the joint flexion score.

[0079] The specific implementation method for the above steps in this embodiment is as follows:

[0080] Based on the recognized image, the positions of various parts of the hand can be determined, thereby establishing three-dimensional hand data including the coordinates of the hand joints. Based on the hand joint coordinates, the length of the hand bones between the j-th and (j+1)-th joints at time t needs to be calculated first. :

[0081]

[0082] in, Let x be the x-coordinate of the (j+1)th joint at time t. Let be the x-coordinate of the j-th joint at time t. Let be the y-coordinate of the (j+1)th joint at time t. Let be the y-coordinate of the j-th joint at time t. Let be the z-axis coordinate of the (j+1)th joint at time t. Let be the z-axis coordinate of the (j+1)th joint at time t. Then, calculate the bone vector between the j-th and (j+1)-th joints at time t. :

[0083] .

[0084] Next, determine the theoretical maximum angle of the joint to be examined when performing the specified gesture. In this embodiment, the index finger after the gesture has stabilized during execution is the primary focus of consideration. The index finger includes the metacarpophalangeal joint closest to the palm, the proximal interphalangeal joint in the middle, and the distal interphalangeal joint furthest away. This allows us to obtain the skeletal vectors between the proximal interphalangeal joint and the metacarpophalangeal joint. And the skeletal vectors between the proximal and distal interphalangeal joints It can also obtain the bone length between the proximal interphalangeal joint and the metacarpophalangeal joint. and the bone length between the proximal and distal interphalangeal joints. Therefore, the first joint angle θ is calculated:

[0085]

[0086] Next, the flexion score of the first joint is calculated based on the first joint angle and the theoretical maximum angle of the joint. :

[0087]

[0088] Subsequently, the joints involved in calculating the first joint angle are replaced with the palmar joint, the metacarpophalangeal joint closest to the palm, and the proximal interphalangeal joint located in the middle. Using the same method as calculating the first joint flexion score, the second joint angle can be obtained based on the skeletal vectors between the metacarpophalangeal and palmar joints, the skeletal vectors between the metacarpophalangeal and proximal interphalangeal joints, the skeletal lengths between the metacarpophalangeal and palmar joints, and the skeletal lengths between the metacarpophalangeal and proximal interphalangeal joints. This allows for the calculation of the second joint flexion score. The joint flexion score includes the first joint flexion score and the second joint flexion score, and the joint angle includes the first joint angle and the second joint angle.

[0089] Then determine the x-axis coordinate of the thumb tip. The y-axis coordinate of the thumb tip The z-axis coordinate of the thumb tip x-axis coordinate of the index fingertip y-axis coordinate of the index fingertip z-axis coordinate of the index fingertip It is also necessary to determine the theoretical maximum distance between the tips of the thumb and index finger when performing a gesture as the maximum distance between the thumb and index finger. This allows us to calculate the thumb-to-finger distance between the tips of the thumb and index finger after the hand gesture has stabilized. :

[0090]

[0091] This allows for the calculation of thumb alignment scores. :

[0092]

[0093] Finally, finger function scores were obtained through thumb alignment and joint flexion scores. :

[0094] .

[0095] S4: Determine the demonstration gesture data, calculate the hand range of motion score and gesture speed score, calculate the gesture accuracy score based on the demonstration gesture data, and calculate the gesture score based on the hand range of motion score, gesture accuracy score, and gesture speed score;

[0096] Furthermore, the objective of this stage is to calculate the hand range of motion score, gesture accuracy score, and gesture speed score to obtain a gesture score. Specifically, step S4 further includes:

[0097] S41: Determine the demonstration gesture data. Theoretical maximum range of motion and theoretical maximum joint velocity are determined. The joint displacement range is obtained using the three-dimensional hand data. A hand range of motion score is calculated based on the joint displacement range and the theoretical maximum range of motion.

[0098] S42: Calculate the joint movement speed, and obtain the gesture speed score based on the joint movement speed and the theoretical maximum joint movement speed;

[0099] S43: Obtain the standard joint position from the demonstration gesture data, calculate the motion accuracy based on the standard joint position, obtain the gesture accuracy score through the motion accuracy, and take the average score of the hand range of motion score, gesture accuracy score and gesture speed score as the gesture score.

[0100] The specific implementation method for the above steps in this embodiment is as follows:

[0101] First, we need to determine the demonstration gesture data, which is the position of each joint in the hand when the gesture is executed correctly. Based on the demonstration gesture data, we can obtain the coordinates of the nth joint after the gesture is executed, which is the standard joint position. In addition, it is necessary to determine the theoretical maximum range of movement. and theoretical maximum joint displacement speed Next, two consecutive frames of 3D hand data during the peak phase of the gesture are taken. Between the two frames, the modulus of the coordinate difference between the joints with the largest positional change compared to the previous frame is subtracted from the modulus of the coordinate difference between the joints with the smallest positional change. This difference yields the joint displacement range R. The hand range of motion score can then be calculated. :

[0102] .

[0103] Based on the three-dimensional data of the hand after the gesture has stabilized, the position coordinates of the nth joint after the gesture is executed can also be obtained. This allows us to calculate the motion accuracy A:

[0104]

[0105] Where N represents the total number of joints. A gesture accuracy score can be obtained based on the precision of the movement. :

[0106]

[0107] in, This represents the maximum allowable error range.

[0108] Take two consecutive frames of 3D hand data during the peak phase of the gesture execution. Between the two frames, take the magnitude of the coordinate difference between the joints with the largest positional change between the two frames as the joint movement distance. Divide this distance by the time difference between the two frames to obtain the joint movement velocity V. Based on the joint movement velocity and the theoretical maximum joint movement velocity, the gesture speed score can be obtained. :

[0109]

[0110] The final hand gesture score is obtained by averaging the scores for hand range of motion, gesture accuracy, and gesture speed. .

[0111] S5: Weighted summation of finger function score and gesture score to obtain hand activity score, and assessment of hand rehabilitation level based on hand activity score.

[0112] Furthermore, the objective of this stage is to obtain a hand activity score through weighted summation, and to assess the degree of hand rehabilitation based on the hand activity score. Specifically, in step S5, multiple typical gestures are identified, hand movement images are acquired when performing the typical gestures, and steps S1 to S4 are executed to obtain multiple finger function scores and gesture scores. Ridge regression is performed on the multiple finger function scores and gesture scores to obtain a weight matrix, and the hand activity score is obtained by weighted summation through the weight matrix.

[0113] The specific implementation method for the above steps in this embodiment is as follows:

[0114] Here, several typical gestures need to be identified. In this embodiment, the typical gestures include clenched fist, OK, hooked hand, pinched hand, scissor hand, slap, gesture eight, two-finger pinch, five-finger spread, thumb up, gesture six, extended index finger, clenched fist with wrist flexion, and clenched fist with wrist extension. Performing each gesture yields a corresponding hand movement image, thus obtaining the corresponding finger function score and gesture score. That is, the gesture score for the q-th typical gesture. Finger function score for the qth typical gesture Ridge regression is performed on the obtained finger function scores and gesture scores to obtain the weight matrix of the q-th typical gesture. To balance the score differences caused by the varying difficulty and range of motion of different gestures, a total hand activity score is obtained.

[0115]

[0116] Q represents the total number of typical hand gestures. The hand activity score allows for the assessment of hand dexterity and the degree of hand recovery in injured individuals.

[0117] The present invention provides a hand rehabilitation assessment device based on gesture scores. The hand rehabilitation assessment device based on gesture scores described below can be referred to in correspondence with the hand rehabilitation assessment method based on gesture scores described above.

[0118] Figure 2 An example is a schematic diagram of the structure of a hand rehabilitation assessment system based on gesture scores, such as... Figure 2 As shown, a hand rehabilitation assessment method based on gesture scores, as described above, includes:

[0119] Spatiotemporal convolution module 100: used to acquire hand action images, extract hand action images to obtain spatial input features and temporal input features, perform spatial graph convolution on the spatial input features to obtain spatial convolution output, and perform temporal graph convolution on the temporal input features to obtain temporal convolution output;

[0120] Image recognition module 200: It is used to perform average pooling and full connection on the spatial convolution output and temporal convolution output to obtain spatiotemporal attention output, divide the spatiotemporal attention output into stages to obtain stage output, and perform multi-scale residual map convolution and global average pooling on the stage output to obtain the recognition image.

[0121] Finger function scoring module 300: used to obtain three-dimensional hand data by recognizing images, calculate joint angles based on the three-dimensional hand data, calculate the thumb-to-finger distance, and calculate the finger function score based on the joint angles and the thumb-to-finger distance;

[0122] Gesture scoring module 400: used to determine demonstration gesture data, calculate hand range of motion score and gesture speed score, calculate gesture accuracy score based on demonstration gesture data, and calculate gesture score based on hand range of motion score, gesture accuracy score and gesture speed score;

[0123] Hand Activity Scoring Module 500: This module is used to weight and sum the finger function score and gesture score to obtain the hand activity score, which is then used to assess the degree of hand rehabilitation.

[0124] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call a computer program in the memory 830 to execute a hand rehabilitation assessment method based on gesture scores, the method including:

[0125] S1: Acquire hand motion images, extract spatial and temporal input features from the hand motion images, perform spatial graph convolution on the spatial input features to obtain spatial convolution output, and perform temporal graph convolution on the temporal input features to obtain temporal convolution output;

[0126] S2: Perform average pooling and full connection on the spatial convolution output and temporal convolution output to obtain the spatiotemporal attention output. Divide the spatiotemporal attention output into stages to obtain the stage output. Perform multi-scale residual map convolution and global average pooling on the stage output to obtain the recognition image.

[0127] S3: Obtain three-dimensional hand data by recognizing images, calculate joint angles and thumb-to-finger distance based on the three-dimensional hand data, and calculate finger function score based on joint angles and thumb-to-finger distance;

[0128] S4: Determine the demonstration gesture data, calculate the hand range of motion score and gesture speed score, calculate the gesture accuracy score based on the demonstration gesture data, and calculate the gesture score based on the hand range of motion score, gesture accuracy score, and gesture speed score;

[0129] S5: Weighted summation of finger function score and gesture score to obtain hand activity score, and assessment of hand rehabilitation level based on hand activity score.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hand rehabilitation assessment method based on gesture scores, characterized in that, include: Step S1: Acquire hand motion images, extract spatial input features and temporal input features from the hand motion images, perform spatial graph convolution on the spatial input features to obtain spatial convolution output, and perform temporal graph convolution on the temporal input features to obtain temporal convolution output; Step S2: Perform average pooling and full connection on the spatial convolution output and temporal convolution output to obtain the spatiotemporal attention output. Divide the spatiotemporal attention output into stages to obtain stage outputs. Perform multi-scale residual map convolution and global average pooling on the stage outputs to obtain the recognition image. Step S3: Obtain three-dimensional hand data by recognizing the image, calculate joint angles and thumb-to-finger distance based on the three-dimensional hand data, and calculate finger function score based on joint angles and thumb-to-finger distance; Step S4: Determine the demonstration gesture data, calculate the hand range of motion score and gesture speed score, calculate the gesture accuracy score based on the demonstration gesture data, and calculate the gesture score based on the hand range of motion score, gesture accuracy score, and gesture speed score; Step S5: Weight the finger function score and gesture score to obtain the hand activity score, and assess the degree of hand rehabilitation based on the hand activity score.

2. The hand rehabilitation assessment method based on gesture scores according to claim 1, characterized in that, In step S1, spatial features are extracted from the hand motion image to obtain the spatial input features, and temporal features are extracted from the hand motion image to obtain the temporal input features.

3. The hand rehabilitation assessment method based on gesture scores according to claim 1, characterized in that, In step S1, the spatial adjacency matrix and the temporal adjacency matrix are determined. The spatial input features are convolved with the spatial graph using the spatial adjacency matrix to obtain the spatial convolution output. The temporal input features are convolved with the temporal graph using the temporal adjacency matrix to obtain the temporal convolution output.

4. The hand rehabilitation assessment method based on gesture scores according to claim 1, characterized in that, In step S2, spatial average pooling is performed on the spatial convolution output to obtain a spatial average pooled output, temporal average pooling is performed on the temporal convolution output to obtain a temporal average pooled output, a fully connected layer is formed on the spatial average pooled output and the temporal average pooled output to obtain a fully connected output, a weighted outer product is performed on the fully connected output to obtain a spatiotemporal attention matrix, and the spatiotemporal attention output is obtained through the spatiotemporal attention matrix.

5. The hand rehabilitation assessment method based on gesture scores according to claim 1, characterized in that, In step S2, the action stage is determined, the feature dimension is determined based on the action stage, the spatiotemporal attention output is divided into stages based on the feature dimension and the action stage to obtain the stage output, and the stage output is subjected to multi-scale residual map convolution and global average pooling to obtain the recognition image.

6. The hand rehabilitation assessment method based on gesture scores according to claim 1, characterized in that, Step S3 further includes: S31: Obtain the three-dimensional data of the hand, including the hand joint coordinates, based on the recognized image, and calculate the hand bone length and bone vector based on the hand joint coordinates; S32: Determine the theoretical maximum angle of the joint, calculate the joint angle based on the bone vector and the length of the hand bones, and obtain the joint flexion score based on the joint angle and the theoretical maximum angle of the joint; S33: Determine the maximum distance between thumb and index finger, calculate the distance between thumb and index finger based on the hand joint coordinates, calculate the thumb alignment score based on the maximum distance between thumb and index finger and the distance between thumb and index finger, and obtain the finger function score through the thumb alignment score and the joint flexion score.

7. The hand rehabilitation assessment method based on gesture scores according to claim 1, characterized in that, Step S4 further includes: S41: Determine the demonstration gesture data, theoretical maximum range of motion, and theoretical maximum joint speed; obtain the joint displacement range through the three-dimensional hand data; and calculate the hand range of motion score based on the joint displacement range and the theoretical maximum range of motion. S42: Calculate the joint movement speed, and obtain the gesture speed score based on the joint movement speed and the theoretical maximum joint movement speed; S43: Obtain the standard joint position from the demonstration gesture data, calculate the motion accuracy based on the standard joint position, obtain the gesture accuracy score through the motion accuracy, and take the average score of the hand range of motion score, gesture accuracy score and gesture speed score as the gesture score.

8. The hand rehabilitation assessment method based on gesture scores according to claim 1, characterized in that, In step S5, multiple typical gestures are identified, hand motion images are obtained when performing typical gestures, and steps S1 to S4 are executed to obtain multiple finger function scores and gesture scores. Ridge regression is performed on the multiple finger function scores and gesture scores to obtain a weight matrix. The hand activity score is obtained by weighted summation through the weight matrix.

9. A hand rehabilitation assessment system based on gesture scores, used to perform a hand rehabilitation assessment method based on gesture scores as described in any one of claims 1 to 8, characterized in that, include: Spatiotemporal convolution module: used to acquire hand action images, extract spatial input features and temporal input features from the hand action images, perform spatial graph convolution on the spatial input features to obtain spatial convolution output, and perform temporal graph convolution on the temporal input features to obtain temporal convolution output; Image recognition module: It is used to perform average pooling and full connection on the spatial convolution output and temporal convolution output to obtain spatiotemporal attention output, divide the spatiotemporal attention output into stages to obtain stage output, and perform multi-scale residual map convolution and global average pooling on the stage output to obtain the recognition image. Finger function scoring module: used to obtain three-dimensional hand data by recognizing images, calculate joint angles and thumb-to-finger distance based on the three-dimensional hand data, and calculate finger function score based on joint angles and thumb-to-finger distance; Gesture scoring module: used to determine demonstration gesture data, calculate hand range of motion score and gesture speed score, calculate gesture accuracy score based on demonstration gesture data, and calculate gesture score based on hand range of motion score, gesture accuracy score and gesture speed score; Hand activity scoring module: This module is used to weight and sum the finger function score and gesture score to obtain a hand activity score, which is then used to assess the degree of hand rehabilitation.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a hand rehabilitation assessment method based on gesture scores as described in any one of claims 1 to 8.

Images