Gesture recognition methods, devices, and media based on YOLOv5-FGS and Mediapipe
By using the YOLOv5-FGS and Mediapipe gesture recognition methods, gesture detection and analysis are processed separately. The improved YOLOv5-FGS model and Mediapipe are used to detect key hand points, which solves the problems of difficult deployment of deep learning methods on edge devices and high secondary development costs, and achieves efficient and fast gesture recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YANCHENG TEACHERS UNIV
- Filing Date
- 2023-10-20
- Publication Date
- 2026-07-17
AI Technical Summary
Deep learning-based gesture recognition methods are difficult to deploy on resource-constrained edge devices, and secondary development costs are high. Traditional methods also exhibit poor robustness and accuracy in complex environments.
A gesture recognition method combining YOLOv5-FGS and Mediapipe is adopted, which separates gesture detection and gesture analysis. The improved YOLOv5-FGS hand detection model and Mediapipe are used to detect key hand points. The model is trained using a Naive Bayes classifier to generate a gesture classification model, reducing training time costs.
Without compromising accuracy, it reduces computational load and parameter count, increases detection speed, reduces training time costs, is suitable for resource-constrained edge devices, and supports the rapid addition of new gesture categories.
Smart Images
Figure CN119296163B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a gesture recognition method, device and medium based on YOLOv5-FGS and Mediapipe. Background Technology
[0002] Gesture recognition can be broadly categorized into sensor-based and computer vision-based methods. Sensor-based methods offer higher accuracy but heavily rely on external hardware, requiring significant investment. Due to the high cost of sensor-based methods, many researchers have focused on computer vision-based gesture recognition. Vision-based gesture recognition primarily involves the steps of gesture segmentation, gesture analysis, and gesture recognition.
[0003] Traditional vision-based gesture recognition methods typically first use segmentation algorithms to separate gestures from complex backgrounds, then extract features based on information obtained from image processing algorithms, such as Histogram of Oriented Gradients (HOG), Scale Invariant Feature Transform (SIFT), and Local Binary Pattern Recognition (LBP). Finally, they combine these methods with machine learning classifiers, such as Support Vector Machines (SVM) and Adaboost, for classification. However, these methods exhibit poor robustness and low recognition accuracy in complex backgrounds.
[0004] With the development of computer vision, using deep learning for gesture recognition has become a popular trend. Compared with traditional gesture recognition methods, deep learning-based gesture recognition methods have better robustness and accuracy in complex backgrounds. Although deep learning-based gesture recognition methods have significant advantages over traditional methods, they also have drawbacks such as requiring massive amounts of data, high training costs (long training time and high-end training equipment), and difficulty in deployment on resource-limited edge devices. Furthermore, if the needs for gesture recognition change, the model needs to be retrained, resulting in significant secondary development costs. Summary of the Invention
[0005] Technical Objective: To address the shortcomings of existing technologies, this invention discloses a gesture recognition method, device, and medium based on YOLOv5-FGS and Mediapipe. By separating gesture detection and gesture analysis, the performance of the detection model and classification model can be improved respectively. If new gestures need to be added later, there is no need to retrain the hand detection. It is only necessary to use the trained YOLOv5-FGS hand detection model to perform hand target detection on the new gesture classification dataset once to locate the effective gesture position, which can effectively reduce the training time cost.
[0006] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution.
[0007] A gesture recognition method based on YOLOv5-FGS and Mediapipe includes the following steps:
[0008] Step S1: Construct a hand detection dataset and train the improved YOLOv5-FGS hand detection model;
[0009] Step S2: Use the YOLOv5-FGS hand detection model to detect hands in the gesture classification dataset, use Mediapipe to detect hand key points, and combine the hand key point information output by Mediapipe to train the Naive Bayes classifier to generate a gesture classification model.
[0010] Step S3: For the video image to be detected, first use the YOLOv5-FGS hand detection model obtained in step S1 to perform single-frame detection and locate the hand position. Then use Mediapipe to detect hand key points. The processed hand key point features are detected by the gesture classification model to output the final gesture target localization result and classification result.
[0011] Preferably, in step S1, the hand detection dataset construction process includes: processing an image dataset containing 18 common gestures and one irrelevant gesture to obtain a binary classification dataset containing only relevant and irrelevant gestures, with a ratio of relevant to irrelevant gestures of 3:1; processing the binary classification dataset to adjust the images in the binary classification dataset to 640*640 pixels to obtain the final hand detection dataset.
[0012] Preferably, in step S1, the improved YOLOv5-FGS hand detection model structure includes: first, reconstructing the C3 model in the original YOLOv5 network using the FastNet network to obtain the C3fast model, and replacing the C3 module in the original YOLOv5 network with C3fast; then, replacing the CBS module of the YOLOv5 network (except for the CBS module in the first layer) with the Ghost convolution module; and finally, adding the SE attention mechanism module to the end of the YOLOv5 feature extraction network.
[0013] Preferably, in step S2, the hand key point feature processing process includes: Mediapipe performs hand key point detection to obtain 21 hand key point coordinates, including the wrist point and 4 joint points corresponding to each of the five fingers. Based on the 21 hand key point coordinates, 5 sets of angle features and 5 sets of distance features are extracted as the basis for judging the bending of the fingers, and 4 sets of angle features are extracted as the basis for judging the opening and closing of the fingers, for a total of 14 sets of features. Then, a threshold is set to binarize the 14 sets of features to obtain the hand key point features.
[0014] Preferably, the 5 sets of angular features correspond to the bending angle θ, the 5 sets of distance features correspond to the modulus ratio d, and the 4 sets of angular features correspond to the opening angle a. The bending angle θ is set to 160 degrees, and values less than or equal to it are converted to 1, while values greater than it are converted to 0. The opening angle a is set to 30 degrees, and values greater than it are converted to 1, while values less than or equal to it are converted to 0. The modulus ratio d is set to a length of 1, and values greater than it are converted to 1, while values less than or equal to it are converted to 0. Finally, a 14-dimensional feature vector with feature values of 0 or 1 is obtained, which serves as the key feature of the hand.
[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a gesture recognition method based on YOLOv5-FGS and Mediapipe as described above.
[0016] A computer-readable storage medium storing computer-executable instructions for performing any of the above-described gesture recognition methods based on YOLOv5-FGS and Mediapipe.
[0017] Beneficial effects: This invention separates gesture detection and gesture analysis, which can improve the performance of the detection model and the classification model respectively. If new gestures need to be added later, there is no need to retrain the hand detection. It is only necessary to use the trained YOLOv5-FGS hand detection model to perform hand target detection on the new gesture classification dataset once to locate the effective gesture position, which can effectively reduce the training time cost. Attached Figure Description
[0018] Figure 1 This is a flowchart of the overall method of the present invention;
[0019] Figure 2 This is an example of an image dataset for the present invention;
[0020] Figure 3 This is a schematic diagram of the improved YOLOv5-FGS hand detection model of the present invention;
[0021] Figure 4 for Figure 3 Schematic diagram of the C3Fast module structure;
[0022] Figure 5 for Figure 3 Schematic diagram of Ghost convolutional module;
[0023] Figure 6 This is a comparison chart of mAP between the existing YOLOv5 network model and the improved YOLOv5-FGS network model in this invention.
[0024] Figure 7 This is a schematic diagram of the image detection results for a portion of the test set of this invention;
[0025] Figure 8 This is an example of the gesture classification dataset of the present invention;
[0026] Figure 9 This is a schematic diagram showing the key hand points of the present invention;
[0027] Figure 10 This is a schematic diagram illustrating the degree of thumb bending according to the present invention;
[0028] Figure 11 This is a schematic diagram illustrating the opening and closing of the thumb and middle finger according to the present invention;
[0029] Figure 12 This is the confusion matrix for the gesture recognition classification results of the present invention. Detailed Implementation
[0030] The following description, in conjunction with the accompanying drawings, further illustrates and explains a gesture recognition method based on YOLOv5-FGS and Mediapipe according to the present invention.
[0031] As attached Figure 1 As shown, a gesture recognition method based on YOLOv5-FGS and Mediapipe includes the following steps:
[0032] Step S1: Construct a hand detection dataset and train the improved YOLOv5-FGS hand detection model;
[0033] Step S2: Use the YOLOv5-FGS hand detection model to detect hands in the gesture classification dataset, use Mediapipe to detect hand key points, and combine the hand key point information output by Mediapipe to train the Naive Bayes classifier to generate a gesture classification model.
[0034] Step S3: For the video image to be detected, first use the YOLOv5-FGS hand detection model obtained in step S1 to perform single-frame detection and locate the hand position. Then use Mediapipe to detect hand key points. The processed hand key point features are detected by the gesture classification model to output the final gesture target localization result and classification result.
[0035] In step S1, the hand detection dataset construction process includes: processing an image dataset containing 18 common gestures and one irrelevant gesture. An example of the image dataset is shown below. Figure 2 As shown, a binary classification dataset containing only relevant and irrelevant gestures is obtained. The relevant gestures are 18 common gestures, and the irrelevant gestures are natural hand movements and postures. The ratio of relevant to irrelevant gestures is 3:1. The binary classification dataset is processed, including reducing the resolution and scaling down proportionally, to adjust the images in the binary classification dataset to 640*640 pixels, resulting in the final hand detection dataset.
[0036] In step S1, the improved YOLOv5-FGS hand detection model structure includes: first, reconstructing the C3 model in the original YOLOv5 network using the FastNet network to obtain the C3fast model, and replacing the C3 module in the original YOLOv5 network with the C3fast model; then, replacing the CBS module of the YOLOv5 network (except for the CBS module in the first layer) with the Ghost convolution module; and finally, adding the SE attention mechanism module to the end of the YOLOv5 feature extraction network.
[0037] The YOLOv5-FGS network described in this invention is illustrated in the following schematic diagram. Figure 3 As shown, there are three improvements compared to the original YOLOv5 network. First, the C3 model in the original YOLOv5 network is reconstructed using the FastNet network to obtain the C3Fast model, where the C3Fast module is as follows: Figure 4 As shown, the C3 module in the original YOLOv5 network is replaced with C3Fast; secondly, the CBS module (except for the CBS module in the first layer) of the YOLOv5 network is replaced with the Ghost convolutional module. A schematic diagram of the Ghost convolutional module is shown below. Figure 5 As shown; thirdly, the SE attention mechanism module is introduced into the end of the YOLOv5 feature extraction network.
[0038] To verify the effectiveness of each improved module, four ablation experiments were conducted in some embodiments of the present invention. The training was performed sequentially, and the best weight file of the training output was tested on the test set. The epoch of each experiment was set to 100, and the initial learning rate was 0.001. The experimental data obtained are shown in Table 1.
[0039] Table 1 Comparison of ablation test performance charts
[0040]
[0041] In Table 1, a checkmark indicates that the modification was performed, and an X indicates that the modification was not performed. The data in Table 1 shows that after replacing the C3 module with C3Fast, mAP improved by 0.2%, computational cost decreased by 20%, the number of parameters decreased by 18%, the weight file size decreased by 27%, and detection time decreased by 0.5ms. Replacing the CBS module with GhostConv resulted in another 0.2% improvement in mAP, a significant decrease in the number of parameters and computational cost, a further reduction in the weight file size, and a slight decrease in detection time. Finally, introducing the SE attention mechanism kept the computational cost unchanged, did not increase detection time, only slightly increased the number of parameters, increased the weight file size by only 0.07MB, and further improved mAP. Compared to the original model, the final model reduced the number of parameters by 34.5%, computational cost by 34.9%, weights by 33.2%, and detection time by 0.7ms. Among them, the average precision of all classes (mAP) represents the mean of the area enclosed by the PR curve and the coordinate axis for each class. It is a comprehensive evaluation of the performance of P and R. mAP@0.5 represents the mAP when the intersection-union ratio (IOU) is 0.5.
[0042] To more intuitively demonstrate the performance of the improved model, Figure 6 The diagram shows a comparison of mAP between the original model (the existing YOLOv5 network model) and the improved YOLOv5 model (the YOLOv5-FGS network model in this invention) during the training process.
[0043] Figure 7 This is a detection result diagram of a portion of images on the final improved YOLOv5-FGS network model in one embodiment of the present invention. It can be seen that all relevant gestures and irrelevant hands can be correctly located (regardless of the target size), and the confidence of the relevant gestures of particular interest is above 90%.
[0044] Step S2 includes: using the YOLOv5-FGS hand detection model to classify a gesture dataset containing 11 types of gestures (example dataset shown). Figure 8 (As shown) Hand detection is performed to locate relevant and irrelevant gestures. Then, Mediapipe is used to detect hand key points of the located relevant gestures. Based on the obtained hand key point information, feature extraction and processing are performed. The final hand key point features and corresponding gesture labels are input into the Naive Bayes classifier for training to generate a gesture classification model.
[0045] In step S2, the hand key point feature processing process includes: as follows Figure 9 As shown, Mediapipe detects 21 hand key points, including the wrist point and four joint points corresponding to each of the five fingers. Based on the coordinates of the 21 hand key points, five sets of angle features and five sets of distance features are extracted as the criteria for judging the bending of the fingers, and four sets of angle features are extracted as the criteria for judging the opening and closing of the fingers, for a total of 14 features. Then, a threshold is set to binarize the 14 features to obtain the hand key point features.
[0046] like Figure 9 As shown, the 21 key points of the hand include the wrist point (point 0), the midpoint between the wrist and the base of the thumb (point 1), the base of the thumb (point 2), the thumb knuckle point (point 3), the tip of the thumb (point 4), the base of the index finger (point 5), the second knuckle point of the index finger (point 6), the first knuckle point of the index finger (point 7), the tip of the index finger (point 8), the base of the middle finger (point 9), the second knuckle point of the middle finger (point 10), the first knuckle point of the middle finger (point 11), the tip of the middle finger (point 12), the base of the ring finger (point 13), the second knuckle point of the ring finger (point 14), the first knuckle point of the ring finger (point 15), the tip of the ring finger (point 16), the base of the little finger (point 17), the second knuckle point of the little finger (point 18), the first knuckle point of the little finger (point 19), and the tip of the little finger (point 20). Five sets of angular features correspond to the bending angle θ, five sets of distance features correspond to the modulus ratio d, and four sets of angular features correspond to the opening angle a. Their relationship with 21 key hand points is shown in Table 2.
[0047] Table 214: Relationship between characteristics and 21 key hand points
[0048]
[0049] To determine the degree of finger bending, we will take the thumb bending as an example. Figure 10 (a)(b)(c) are schematic diagrams showing the degree of bending of the thumb. Figure 10 (a) shows the result of the model recognizing 21 key hand points. Figure 10 (b) shows a diagram illustrating the distance ratio used to determine the bending of the thumb. In the diagram, the base of the little finger (point 17) and the base of the thumb (point 2) are vectors A (17, 2), and the base of the little finger (point 17) and the apex of the thumb (point 4) are vectors B (17, 4). The modulus ratio d of the two vectors can reflect the bending of the thumb. The calculation method is shown in equation (1):
[0050]
[0051] Where x2 and y2 are the coordinates of the base of the thumb (point 2), x 17 y 17x4 and y4 are the coordinates of the base of the little finger (point 17), and x4 and y4 are the coordinates of the apex of the thumb (point 4).
[0052] Figure 10 (c) shows a schematic diagram of the angle for judging the bending of the thumb. In the diagram, the base of the thumb (point 2) and the wrist point (point 0) are vectors C(2, 0), and the base of the thumb (point 2) and the tip of the thumb (point 4) are vectors D(2, 4). The degree of the angle θ between the two vectors can reflect the degree of bending of the thumb. The calculation formula is shown in equation (2):
[0053]
[0054] Where x2 and y2 are the coordinates of the base of the thumb (point 2), x4 and y4 are the coordinates of the tip of the thumb (point 4), and x0 and y0 are the coordinates of the wrist point (point 0).
[0055] Judging the opening and closing of the fingers: Here we take the opening and closing of the index and middle fingers as an example, such as... Figure 11 (a) and (b) show schematic diagrams illustrating the angles used to determine the opening and closing of the index and middle fingers. Figure 11 (a) shows the result of the model recognizing 21 key hand points. Figure 11 (b) shows a schematic diagram of the angle for judging the opening and closing of the index and middle fingers. In the diagram, the base of the index finger (point 5) and the tip of the index finger (point 8) are vector A (5, 8), and the base of the index finger (point 5) and the tip of the middle finger (point 12) are vector B (5, 12). The degree of the angle α between the two vectors can reflect the degree of bending of the thumb. The calculation method is the same as the above formula (2).
[0056] The calculation method for the remaining finger bending and finger opening / closing features is the same as above. After calculation, 14 sets of feature values can be obtained. Since the extracted geometric features are divided into angles and distance ratios, which have different physical meanings and large differences in value range, the complexity is high. Here, based on setting a certain threshold, the extracted angle and distance ratio features are converted into binary features.
[0057] Five sets of angular features correspond to the bending angle θ, five sets of distance features correspond to the modulus ratio d, and four sets of angular features correspond to the opening angle a. In some embodiments of the present invention, the bending angle θ is set to 160 degrees, and values less than or equal to it are converted to 1, while values greater than it are converted to 0. The opening angle a is set to 30 degrees, and values greater than it are converted to 1, while values less than or equal to 30 degrees are converted to 0. The modulus ratio d is set to a length of 1, and values greater than it are converted to 1, while values less than or equal to it are converted to 0. After the above processing, a 14-dimensional feature vector with feature values of 0 or 1 is obtained.
[0058] Step S3 includes: using the YOLOv5-FGS hand detection model to detect the detected single-frame video image, locating relevant and irrelevant gestures; then using Mediapipe to detect hand keypoints of the located relevant gestures; extracting and processing features based on the obtained hand keypoint information; and inputting the final hand keypoint features into the gesture classification model trained in step S2 to obtain the final gesture target localization result and classification result. The confusion matrix of the final gesture recognition classification result is as follows: Figure 12 As shown.
[0059] This invention also discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements one of the gesture recognition methods based on YOLOv5-FGS and Mediapipe described above. The memory can be of various types, such as random access memory, read-only memory, flash memory, etc. The processor can be of various types, such as a central processing unit, microprocessor, digital signal processor, or image processor.
[0060] This invention also discloses a computer-readable storage medium storing computer-executable instructions for executing any of the gesture recognition methods based on YOLOv5-FGS and Mediapipe described above. The storage medium includes various media capable of storing program code, such as USB flash drives, external hard drives, ROM, RAM, magnetic disks, or optical disks.
[0061] This invention integrates the advantages of deep learning and traditional methods while addressing their shortcomings. First, the original YOLOv5 network is lightweighted to obtain YOLOv5-FGS. Without affecting its accuracy, this significantly reduces the computational cost and number of parameters, accelerates the inference speed of the network model, and reduces the memory usage, making it more suitable for deployment on resource-constrained edge devices. Second, a binary hand detection model is creatively trained using the YOLOv5-FGS network to exclude irrelevant gestures and locate relevant gestures. This model can serve as a general hand detection model, adding gesture categories. The dataset only needs to be processed once using the general hand detection model, and then a new gesture classification model can be quickly trained using traditional machine learning classification methods (requiring only a small number of samples). This effectively solves the problem of high secondary development costs for deep learning-based gesture recognition methods. Finally, a Mediapipe-based hand keypoint detection method is introduced into the feature extraction stage of the classification model, greatly improving classification accuracy.
[0062] Furthermore, the method described in this invention separates gesture detection and gesture analysis, which can improve the performance of the detection model and the classification model respectively. If new gestures need to be added later, there is no need to retrain the hand detection. It is only necessary to use the trained YOLOv5-FGS hand detection model to perform hand target detection on the new gesture classification dataset once to locate the effective gesture position. This can effectively reduce the training time cost. It can be used in different scenarios in gesture recognition tasks and has a good application prospect.
[0063] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A gesture recognition method based on YOLOv5-FGS and Mediapipe, characterized in that, Includes the following steps: Step S1: Construct a hand detection dataset and train the improved YOLOv5-FGS hand detection model; In step S1, the improved YOLOv5-FGS hand detection model structure includes: first, reconstructing the C3 module in the original YOLOv5 network using the FastNet network to obtain the C3fast model, and replacing the C3 module in the original YOLOv5 network with the C3fast model; then, replacing the CBS module of the YOLOv5 network with the Ghost convolutional module; and finally, adding the SE attention mechanism module to the end of the YOLOv5 feature extraction network. Step S2: The YOLOv5-FGS hand detection model is used to detect hands on the gesture classification dataset. Mediapipe is used for hand keypoint detection. The Naive Bayes classifier is trained using the hand keypoint information output by Mediapipe to generate a gesture classification model. In step S2, the hand keypoint feature processing includes: Mediapipe detects 21 hand keypoint coordinates, including the wrist point and four joints corresponding to each of the five fingers. Based on these 21 hand keypoint coordinates, five sets of angle features and five sets of distance features are extracted as criteria for determining finger bending. Four sets of angle features are extracted as criteria for determining finger opening and closing, resulting in a total of 14 sets of features. A threshold is then set to binarize these 14 sets of features to obtain the hand keypoint features. The five sets of angle features correspond to bending angles. Five sets of distance features correspond to the modulus ratio d, and four sets of angle features correspond to the opening angle a and bending angle. Using 160 degrees as the threshold, the bending angle When the angle is less than or equal to 160 degrees, the bending angle characteristic value is converted to 1, and the bending angle... When the angle is greater than 160 degrees, the bending angle feature value is converted to 0. The opening angle 'a' is set to a threshold of 30 degrees. When the opening angle 'a' is greater than 30 degrees, the opening angle feature value is converted to 1. When the opening angle 'a' is less than or equal to 30 degrees, the opening angle feature value is converted to 0. The modulus ratio 'd' is set to a threshold of 1. When the modulus ratio 'd' is greater than 1, the modulus ratio feature value is converted to 1. When the modulus ratio 'd' is less than or equal to 1, the modulus ratio feature value is converted to 0. Finally, a 14-dimensional feature vector with feature values of 0 or 1 is obtained, which serves as the key feature of the hand. Step S3: For the video image to be detected, first use the YOLOv5-FGS hand detection model obtained in step S1 to perform single-frame detection and locate the hand position. Then use Mediapipe to detect hand key points. The processed hand key point features are then detected by the gesture classification model to output the final gesture target localization result and classification result.
2. The gesture recognition method based on YOLOv5-FGS and Mediapipe according to claim 1, characterized in that, In step S1, the hand detection dataset construction process includes: processing an image dataset containing 18 common gestures and one irrelevant gesture to obtain a binary classification dataset containing only relevant and irrelevant gestures, with a ratio of relevant to irrelevant gestures of 3:1; processing the binary classification dataset to adjust the images in the binary classification dataset to 640*640 pixels to obtain the final hand detection dataset.
3. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a gesture recognition method based on YOLOv5-FGS and Mediapipe as described in claim 1 or 2.
4. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for performing a gesture recognition method based on YOLOv5-FGS and Mediapipe as described in claim 1 or 2.