Dynamic gesture recognition method based on hand skeleton data

Abstract

The invention relates to a dynamic gesture recognition method based on hand skeleton data, and belongs to the field of medicine, the method comprises the steps of S1, data acquisition and processing, S2, dynamic gesture recognition based on hand skeleton data, S3, C3d model optimization adjustment, C3d model data input size adjustment, and maximum pooling layer addition. The method can better adapt to the recognition requirements of the special gestures of the stroke patient, the size change is matched, and the final feature dimension is not affected. The dynamic gesture recognition based on the hand skeleton data plays a very important role by adjusting the size of the input data, as the features contained in the hand skeleton data are much simpler than those contained in RGB data, the model calculation amount is greatly reduced, and redundant information in the aspects of background, illumination change, hand appearance and the like is also eliminated, so that the dynamic gesture recognition accuracy is greatly improved. And the model can focus on key features more quickly, so that the training time is remarkably shortened.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a dynamic gesture recognition method based on hand skeleton data. Background Art

[0002] Stroke is a common neurological disease that occurs frequently in the elderly. 80% of stroke patients have hand dysfunction. Clinically, the hand function of patients will be significantly improved after repeated rehabilitation treatment. Therefore, hand function must be accurately evaluated when formulating rehabilitation plans and observing the efficacy in the future. At present, rehabilitation doctors are usually required to conduct manual evaluation in combination with medical qualitative scales in clinical practice. The results are usually not accurate and are affected by subjective factors.

[0003] At present, there are mainly three types of methods, namely those based on wearable devices, those based on optical capture devices, and those based on computer vision. The rehabilitation training and evaluation methods based on wearable devices can recognize hand movements and spatial three-dimensional information in real time, can recognize more gestures, have high accuracy, are not easily affected by external interference, and have good robustness, but are usually expensive. When patients sweat, accuracy may be affected, and patients may also feel fatigued. In addition, the wearing methods of some rehabilitation equipment such as mechanical gloves have certain limitations for patients with poor hand motor ability, and heavier equipment may also cause secondary injuries to patients. The method based on optical capture devices has basically no interference to users, but Leap Motion can obtain too little information about the side fingers, making it difficult to recognize side gestures. For some more complex movements, the gesture feature points may be blocked, resulting in incorrect recognition of the number of fingers and inaccurate gesture recognition.

[0004] Since stroke patients generally have upper limb weakness and inability to flex and extend their fingers, their specific movements are not standard, the amplitude is small, or the speed is slow. Therefore, it is difficult to correctly identify such abnormal gestures. In addition, there is currently no standardization and normalization for quantitative evaluation methods, so how to reasonably set standards to objectively evaluate the patient's recovery is also one of the difficulties in subsequent evaluations. Summary of the invention

[0005] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides a dynamic gesture recognition based on hand skeleton data, which uses computer vision methods to solve the problem that Leap Motion can only obtain too little side finger information and it is difficult to recognize side gestures. For some more complex movements, gesture feature points may be blocked, resulting in incorrect recognition of the number of fingers and inaccurate gesture recognition.

[0006] (II) Technical solution In order to achieve more accurate and efficient hand function motion recognition, the present invention provides the following technical solution: a dynamic gesture recognition method based on hand skeleton data, comprising the following steps: Step S1, self-build hand function action data set, select nine hand function actions according to FuglMer motor function assessment scale, and collect video data respectively to form hand function data set; Step S2, data processing, using the Mediapipe framework to obtain the coordinates of the key points of the hand, and construct the hand skeleton shape, converting the video dataset into a video dataset containing only the hand skeleton information to eliminate the interference of the background and hand appearance information. The original RGB video data is processed to form three sets of comparative experimental data, including the original RGB video data, the Skeleton video data containing only the hand skeleton information, and the RGB+Skeleton video data with the hand skeleton information superimposed on the original RGB video; Step S3, optimization and adjustment of the C3d model, adjusting the data input size of the C3d model and adding a maximum pooling layer to better meet the recognition requirements of special gestures of stroke patients, so as to match the size change without affecting the final feature dimension; Step S4, dynamic gesture recognition based on hand skeleton data, using hand skeleton data as input, applying the adjusted C3d model to the hand skeleton data set for training and prediction to achieve accurate recognition of gestures. The recognition results are used to evaluate the hand function rehabilitation of stroke patients to ensure the objectivity and accuracy of the evaluation.

[0007] Preferably, in step S1, RGB video data containing nine standard hand movements are collected, namely ulnar deviation, wrist dorsiflexion, wrist palmar flexion, ball grasp, four-finger flexion, forearm pronation, finger adduction and abduction, thumb abduction, and thumb to index finger. The data set contains videos of healthy hands and diseased hands for subsequent gesture recognition and evaluation.

[0008] Preferably, in step S1, in the process of self-building the data set, a total of 2520 RGB video data are collected, 9 types of actions, 280 videos in each type, including 168 healthy hand videos and 112 diseased hand videos, with durations ranging from 2s to 8s.

[0009] Preferably, in step S2, the Mediapipe framework is used to obtain the coordinates of 21 key points of the hand, and the hand skeleton shape is formed by connecting them in the order of the hand joint points. The self-made RGB gesture video dataset is converted into a video dataset containing only hand skeleton information, which is sent to the C3d model for training and prediction, eliminating the influence of background and hand appearance information, and only focusing on the changes caused by the movement process. The processed data forms three groups of comparative experiments.

[0010] Preferably, in step S3, the original data input size of the C3d model is set at 121×121. The collected data needs to be greatly reduced and cropped, which is easy to lose subtle changes in movement. Therefore, in order to retain more information, the input size is changed to 255×255. At the same time, in order to match the size change and not affect the final feature dimension, a maximum pooling layer with a kernel of 1×2×2 is added to the C3d model. The use of maximum pooling can highlight smaller offsets and changes in the data, which is conducive to coping with the particularity of this recognition task.

[0011] Preferably, in step S4, the features contained in the hand skeleton data are simpler than those of the RGB data, the amount of model calculation is greatly reduced, and redundant information is eliminated, so that the model can focus on key features more quickly, thereby significantly reducing training time.

[0012] Preferably, in step S4, when the dynamic gesture recognition model does not use a pre-trained model, the detection accuracy of Skeleton data is improved by at least 1.5% compared with the original RGB data, and the detection accuracy of RGB+Skeleton data is improved by more than 3% compared with the original RGB data.

[0013] (III) Beneficial effects Compared with the prior art, the present invention provides a dynamic gesture recognition based on hand skeleton data, which has the following beneficial effects: The dynamic gesture recognition based on hand skeleton data plays a very important role by adjusting the input data size. Since the features contained in hand skeleton data are much simpler than RGB data, the model calculation amount is greatly reduced, and redundant information such as background, lighting changes, and hand appearance are also excluded. The model can focus on key features more quickly, so the training time is significantly reduced. The highest detection accuracy and shortest training time can be obtained when Skeleton data is used as input, which proves the effectiveness and efficiency of the proposed dynamic gesture recognition method based on hand skeleton data. DETAILED DESCRIPTION

[0014] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] Embodiment 1: A dynamic gesture recognition method based on hand skeleton data comprises the following steps: Step S1, self-build hand function action dataset, collect RGB video data containing nine standard hand actions according to the FuglMer motor function assessment scale, namely ulnar deviation, wrist dorsiflexion, wrist palmar flexion, ball grasp, four-finger flexion, forearm pronation, finger adduction and abduction, thumb abduction, thumb to index finger. The dataset contains videos of healthy hands and diseased hands for subsequent gesture recognition and evaluation. During the data preparation process, a total of 2520 RGB video data were collected, 9 types of actions, 280 videos in each category, including 168 healthy hands and 112 diseased hands videos, with durations ranging from 2s to 8s.

[0016] Step S2, dynamic gesture recognition based on hand skeleton data. The PoseC3d algorithm uses the key points of the human skeleton to form a human skeleton to realize action recognition. The algorithm cannot accurately identify the key points of the hand. Therefore, inspired by it, a dynamic gesture recognition method based on hand skeleton data is designed.

[0017] First, the Mediapipe framework is used to obtain the coordinates of 21 key points of the hand, and the hand skeleton shape is formed by connecting them in the order of the hand joints. The self-made RGB gesture video dataset is converted into a video dataset containing only hand skeleton information, which is sent to the C3d model for training and prediction. This can eliminate the influence of background and hand appearance information and only focus on the changes brought about by the movement process. The original RGB video data is processed to form three sets of comparative experimental data, including the original RGB video data, the Skeleton video data containing only hand skeleton information, and the RGB+Skeleton video data with hand skeleton information superimposed on the original RGB video.

[0018] Step S3, the dynamic gesture recognition part is implemented using the C3d model, using the public Ucf-101 pre-training model, the training results are stable and the recognition accuracy is high. Optimization is performed on the original C3d model, the data input size of the C3d model is adjusted, and the maximum pooling layer is added to better meet the recognition needs of special gestures of stroke patients. To match the size change, the final feature dimension is not affected. The original C3d model's data input size is set at 121×121, and the collected data needs to be greatly reduced and cropped, which is easy to lose subtle movement changes. Therefore, in order to retain more information, the input size is modified to 255×255. At the same time, in order to match the size change and not affect the final feature dimension, a maximum pooling layer with a kernel of 1×2×2 is added to the C3d model. The use of maximum pooling can highlight smaller offsets and changes in the data, which is conducive to coping with the particularity of this recognition task.

[0019] Step S4, using the hand skeleton data as input, the adjusted C3d model is applied to the hand skeleton data set for training and prediction to achieve accurate recognition of gestures. The recognition results are used to evaluate the hand function rehabilitation of stroke patients to ensure the objectivity and accuracy of the evaluation. In step S4, the features contained in the hand skeleton data are simpler than those of the RGB data, the amount of model calculation is greatly reduced, and redundant information is excluded, so that the model can focus on key features more quickly, thereby significantly reducing the training time. In the case of the dynamic gesture recognition model without using a pre-trained model, the detection accuracy of the Skeleton data is improved by at least 1.5% compared to the original RGB data, and the detection accuracy of the RGB+Skeleton data is improved by more than 3% compared to the original RGB data.

[0020] Experimental example: A dynamic gesture recognition method based on hand skeleton data is used for hand function assessment of stroke patients. The data is first processed to form three groups of comparative experiments, with the inputs being original RGB video data, Skeleton video data containing only hand skeleton information, and RGB+Skeleton video data with hand skeleton information superimposed on the original RGB video. After experimental verification, as shown in Table 1, the test accuracy of RGB data is 97.82%, the test accuracy of Skeleton data is 99.01%, and the test accuracy of RGB+Skeleton data is 98.61%, which improves the accuracy by about 1.2%. Due to the effect of the pre-training model, the improvement in accuracy is not significant enough, so a group of experiments is conducted without the pre-training model. The experimental results are shown in Table 2. The detection accuracy of Skeleton is improved by 1.5% compared with the original RGB data, and the detection accuracy of RGB+Skeleton is improved by more than 3% compared with the original RGB data. The adjustment of the input data size also plays a very important role. The experimental results are shown in Table 3, among which the improvement effect is particularly obvious when Skeleton data is input. In addition, since the features contained in hand skeleton data are much simpler than RGB data, the amount of model calculation is greatly reduced, and redundant information such as background, lighting changes, and hand appearance is also excluded. The model can focus on key features more quickly, so the training time is significantly reduced. As shown in Table 1, the highest detection accuracy and shortest training time can be obtained when Skeleton data is used as input, which proves the effectiveness and efficiency of the proposed dynamic gesture recognition method based on hand skeleton data.

[0021]

[0022] The beneficial effect of the present invention is that the dynamic gesture recognition based on hand skeleton data plays a very important role by adjusting the input data size. Since the features contained in the hand skeleton data are much simpler than those in the RGB data, the amount of model calculation is greatly reduced, and redundant information such as background, lighting changes, and hand appearance is also excluded. The model can focus on key features more quickly, so the training time is significantly reduced. The highest detection accuracy and the shortest training time can be obtained when Skeleton data is used as input, which proves the effectiveness and efficiency of the proposed dynamic gesture recognition method based on hand skeleton data.

[0023] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dynamic gesture recognition method based on hand skeleton data, characterized in that: The following steps are involved: Step S1, self-build hand function action data set, select nine hand function actions according to FuglMer motor function assessment scale, and collect video data respectively to form hand function data set; Step S2, data processing, using the Mediapipe framework to obtain the coordinates of the key points of the hand, and construct the hand skeleton shape, converting the video dataset into a video dataset containing only the hand skeleton information to eliminate the interference of the background and hand appearance information. The original RGB video data is processed to form three sets of comparative experimental data, including the original RGB video data, the Skeleton video data containing only the hand skeleton information, and the RGB+Skeleton video data with the hand skeleton information superimposed on the original RGB video; Step S3, optimization and adjustment of the C3d model, adjusting the data input size of the C3d model and adding a maximum pooling layer to better meet the recognition requirements of special gestures of stroke patients, matching the size change without affecting the final feature dimension; Step S4, dynamic gesture recognition based on hand skeleton data, using hand skeleton data as input, applying the adjusted C3d model to the hand skeleton data set for training and prediction to achieve accurate recognition of gestures. The recognition results are used to evaluate the hand function rehabilitation of stroke patients to ensure the objectivity and accuracy of the evaluation.

2. The method for dynamic gesture recognition based on hand skeleton data according to claim 1, characterized in that: In step S1, RGB video data containing nine standard hand movements are collected, namely ulnar deviation, wrist dorsiflexion, wrist palmar flexion, ball grasp, four-finger flexion, forearm pronation, finger adduction and abduction, thumb abduction, and thumb to index finger. The data set contains videos of healthy hands and diseased hands for subsequent gesture recognition and evaluation.

3. The method for dynamic gesture recognition based on hand skeleton data according to claim 1, characterized in that: In step S1, during the data preparation process, a total of 2520 RGB video data were collected, including 9 types of actions, 280 videos in each type, including 168 healthy hand and 112 diseased hand videos, with duration ranging from 2s to 8s.

4. The method for dynamic gesture recognition based on hand skeleton data according to claim 1, characterized in that: In step S2, the Mediapipe framework is used to obtain the coordinates of 21 key points of the hand, and the hand skeleton shape is formed by connecting them in the order of the hand joints. The self-made RGB gesture video dataset is converted into a video dataset containing only hand skeleton information, eliminating the influence of background and hand appearance information, and only focusing on the changes caused by the movement process. The processed data forms three groups of comparative experiments.

5. The method for dynamic gesture recognition based on hand skeleton data according to claim 1, characterized in that: In step S3, the original data input size of the C3d model is set at 121×121. The collected data needs to be greatly reduced and cropped, which is easy to lose subtle movement changes. Therefore, in order to retain more information, the input size is changed to 255×255. At the same time, in order to match the size change and not affect the final feature dimension, a maximum pooling layer with a kernel of 1×2×2 is added to the C3d model. The use of maximum pooling can better highlight smaller offsets and changes in the data, which is conducive to coping with the particularity of this recognition task.

6. According to the method for dynamic gesture recognition based on hand skeleton data, it is characterized in that: In step S4, the features contained in the hand skeleton data are simpler than those in the RGB data, the amount of model calculation is greatly reduced, and redundant information is eliminated, so that the model can focus on key features more quickly, thereby significantly reducing the training time.

7. According to the method for dynamic gesture recognition based on hand skeleton data, it is characterized in that: In step S4, when the dynamic gesture recognition model does not use a pre-trained model, the detection accuracy of the Skeleton data is improved by at least 1.5% compared with the original RGB data, and the detection accuracy of the RGB+Skeleton data is improved by more than 3% compared with the original RGB data.

Images