Gesture recognition method and device based on YOLOv5-FGS and Mediape and medium

Abstract

The invention discloses a gesture recognition method and device based on YOLOv5-FGS and Mediape and a medium. The method comprises the steps that a hand detection data set is constructed, and an improved YOLOv5-FGS hand detection model is obtained through training; performing hand detection on the gesture classification data set by using a YOLOv5-FGS hand detection model, performing hand key point detection through Mediape, training a naive Bayes classifier in combination with hand key point information output by the Mediape, and generating a gesture classification model; according to a video image to be detected, single-frame detection and hand position positioning are firstly carried out through the YOLOv5-FGS hand detection model, then hand key point detection is carried out through Mediape, hand key point features obtained through processing are detected through the gesture classification model, and a final gesture target positioning result and a final gesture target classification result are output.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a gesture recognition method, device and medium based on YOLOv5-FGS and Mediapipe. Background Art

[0002] Gesture recognition can be roughly divided into sensor-based gesture recognition methods and computer vision-based gesture recognition methods. Sensor-based gesture recognition methods have a higher accuracy rate, but they are heavily dependent on external hardware devices and require a lot of investment. Due to the high cost limitation of sensor-based gesture recognition methods, many scholars have begun to focus on computer vision-based gesture recognition methods. Vision-based gesture recognition methods are mainly divided into the steps of segmenting gestures, analyzing gestures, and recognizing gestures.

[0003] Traditional vision-based gesture recognition methods generally use segmentation algorithms to separate gestures from complex backgrounds, and then extract features based on information obtained from image processing algorithms, such as the Histogram of Oriented Gradients (HOG) algorithm, the Scale-Invariant Feature Transform (SIFT) algorithm, the Local Binary Pattern (LBP) algorithm, etc.; and then combine machine learning classifiers, such as the Support Vector Machine (SVM) and Adaptive Boosting (Adaboost), for classification. However, such methods have poor robustness and low recognition accuracy in complex backgrounds.

[0004] With the development of computer vision, the use of deep learning for gesture recognition has become a hot trend. Compared with traditional gesture recognition methods, gesture recognition methods based on deep learning have better robustness and accuracy in complex backgrounds. Although gesture recognition methods based on deep learning have obvious advantages over traditional gesture recognition methods, deep learning has disadvantages such as massive data requirements, high training costs (long training time and high training equipment requirements), and difficulty in deployment on edge devices with limited resources. Once the requirements for gesture recognition change, the model needs to be retrained, and the cost of secondary development is high. Summary of the invention

[0005] Technical purpose: In view of the defects in the prior art, the present invention discloses a gesture recognition method, device and medium based on YOLOv5-FGS and Mediapipe, which separately process gesture detection and gesture analysis, and can respectively improve the performance of the detection model and the classification model in a targeted manner. If new gestures need to be added in the later stage, there is no need to train the hand detection again. It is only necessary to use the trained YOLOv5-FGS hand detection model to perform hand target detection on the new gesture classification data set once to locate the effective gesture position, which can effectively reduce the time cost of training.

[0006] Technical solution: In order to achieve the above technical objectives, the present invention adopts the following technical solution.

[0007] A gesture recognition method based on YOLOv5-FGS and Mediapipe, comprising the following steps:

[0008] Step S1, constructing a hand detection data set and training an improved YOLOv5-FGS hand detection model;

[0009] Step S2: Use the YOLOv5-FGS hand detection model to perform hand detection on the gesture classification dataset, perform hand key point detection through Mediapipe, and train the naive Bayes classifier based on the hand key point information output by Mediapipe to generate a gesture classification model.

[0010] Step S3: For the video image to be detected, first perform single-frame detection through the YOLOv5-FGS hand detection model obtained in step S1 to locate the hand position, and then perform hand key point detection through Mediapipe. The processed hand key point features are detected by the gesture classification model to output the final gesture target positioning result and classification result.

[0011] Preferably, in step S1, the hand detection data set construction process includes: processing an image data set including 18 categories of common gestures and one irrelevant gesture to obtain a binary classification data set containing only relevant gestures and irrelevant gestures, and the ratio of relevant gestures to irrelevant gestures is 3:1; performing data processing on the binary classification data set, adjusting the images in the binary classification data set to images of 640*640 size, and obtaining the final hand detection data set.

[0012] Preferably, in step S1, the improved YOLOv5-FGS hand detection model structure includes: first using the FastNet network to reconstruct the C3 model in the original YOLOv5 network to obtain a C3fast model, and using C3fast to replace the C3 module in the original YOLOv5 network; then using the Ghost convolution module to replace the CBS module of the YOLOv5 network (except the CBS module of the first layer); finally, adding the SE attention mechanism module to the end of the YOLOv5 feature extraction network.

[0013] Preferably, in step S2, the hand key point feature processing process includes: Mediapipe performs hand key point detection to obtain 21 hand key point coordinates, including the wrist point and 4 joint points corresponding to the five fingers, extracts 5 groups of angle features and 5 groups of distance features as a basis for judging the bending of the fingers according to the 21 hand key point coordinates, extracts 4 groups of angle features as a basis for judging the opening and closing of the fingers, and obtains a total of 14 groups of features, and then sets a threshold to perform binarization processing on the 14 groups of features to obtain the hand key point features.

[0014] Preferably, the 5 groups of angle features correspond to the bending angle θ, the 5 groups of distance features correspond to the modulus ratio d, and the 4 groups of angle features correspond to the opening and closing angle a. The bending angle θ takes 160 degrees as the threshold, and is converted to 1 if it is less than or equal to, and is converted to 0 if it is greater than. The opening and closing angle a takes 30 degrees as the threshold, and is converted to 1 if it is greater than, and is converted to 0 if it is less than or equal to 30 degrees. The modulus ratio d takes length 1 as the threshold, and is converted to 1 if it is greater than, and is converted to 0 if it is less than or equal to. Finally, a 14-dimensional feature vector with a eigenvalue of 0 or 1 is obtained as the hand key point feature.

[0015] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned gesture recognition methods based on YOLOv5-FGS and Mediapipe is implemented.

[0016] A computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are used to execute any of the above-mentioned gesture recognition methods based on YOLOv5-FGS and Mediapipe.

[0017] Beneficial effect: The present invention processes gesture detection and gesture analysis separately, which can improve the performance of the detection model and the classification model respectively. If new gestures need to be added in the later stage, there is no need to train hand detection again. It is only necessary to use the trained YOLOv5-FGS hand detection model to perform hand target detection on the new gesture classification data set to locate the effective gesture position, which can effectively reduce the time cost of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a general method flow chart of the present invention;

[0019] Figure 2 is an example of an image dataset of the present invention;

[0020] Figure 3 Schematic diagram of the improved YOLOv5-FGS hand detection model of the present invention;

[0021] Figure 4 for Figure 3 Schematic diagram of the C3Fast module structure;

[0022] Figure 5 for Figure 3 Schematic diagram of Ghost convolution module;

[0023] Figure 6 It is a mAP comparison chart of the existing YOLOv5 network model and the improved YOLOv5-FGS network model in the present invention;

[0024] Figure 7 This is a schematic diagram of the detection results of some test set images of the present invention;

[0025] Figure 8 An example of a gesture classification dataset of the present invention;

[0026] Fig. 9 This is a schematic diagram of the sequence numbers of the key points of the hand of the present invention;

[0027] Fig.10 A schematic diagram of the bending degree of the thumb of the present invention;

[0028] Fig.11 It is a schematic diagram of the opening and closing of the thumb and middle finger of the present invention; Fig.12 It is the confusion matrix of the gesture recognition classification result of the present invention. DETAILED DESCRIPTION

[0029] The following is a further description and explanation of a gesture recognition method based on YOLOv5-FGS and Mediapipe of the present invention in conjunction with the accompanying drawings.

[0030] As attached Figure 1 As shown, a gesture recognition method based on YOLOv5-FGS and Mediapipe includes the following steps:

[0031] Step S1, constructing a hand detection data set and training an improved YOLOv5-FGS hand detection model;

[0032] Step S2: Use the YOLOv5-FGS hand detection model to perform hand detection on the gesture classification dataset, perform hand key point detection through Mediapipe, and train the naive Bayes classifier based on the hand key point information output by Mediapipe to generate a gesture classification model.

[0033] Step S3: For the video image to be detected, first perform single-frame detection through the YOLOv5-FGS hand detection model obtained in step S1 to locate the hand position, and then perform hand key point detection through Mediapipe. The processed hand key point features are detected by the gesture classification model to output the final gesture target positioning result and classification result.

[0034] In step S1, the hand detection data set construction process includes: processing an image data set including 18 types of common gestures and one irrelevant gesture, an example of the image data set is as follows: Figure 2 As shown in the figure, a binary classification data set containing only relevant gestures and irrelevant gestures is obtained, wherein the relevant gestures are 18 categories of common gestures, and the irrelevant gestures are natural hand movements and postures; the ratio of relevant gestures to irrelevant gestures is 3:1, and the binary classification data set is processed, including reducing the resolution, reducing the proportion, etc., and the images in the binary classification data set are adjusted to images of 640*640 size to obtain the final hand detection data set.

[0035] In the step S1, the improved YOLOv5-FGS hand detection model structure includes: firstly reconstructing the C3 model in the original YOLOv5 network using the FastNet network to obtain the C3fast model, and replacing the C3 module in the original YOLOv5 network with the C3fast; then replacing the CBS module of the YOLOv5 network (except the CBS module of the first layer) with the Ghost convolution module; and finally adding the SE attention mechanism module to the end of the YOLOv5 feature extraction network.

[0036] The YOLOv5-FGS network of the present invention, the schematic diagram of the YOLOv5-FGS network structure is as follows Figure 3 As shown in Figure 2, there are three improvements compared to the original YOLOv5 network. First, the C3 model in the original YOLOv5 network is reconstructed using the FastNet network to obtain the C3Fast model, where the C3Fast module is as follows: Figure 4 As shown in the figure, C3Fast is used to replace the C3 module in the original YOLOv5 network; second, the Ghost convolution module is used to replace the CBS module of the YOLOv5 network (except the CBS module of the first layer). The schematic diagram of the Ghost convolution module is shown in Figure 5 As shown; the third is to introduce the SE attention mechanism module into the end of the YOLOv5 feature extraction network.

[0037] In order to verify the effectiveness of each improved module, four ablation experiments were conducted in some embodiments of the present invention. Training was performed in sequence, and the best weight file output by training was taken for testing on the test set. The epoch of each round of experiments was set to 100, and the initial learning rate was 0.001. The obtained experimental data are shown in Table 1:

[0038] Table 1. Comparison of ablation test performance

[0039]

[0040] In Table 1, a check indicates that this modification has been made, and a cross indicates that this modification has not been made. From the data in Table 1, it can be seen that after replacing the C3 module with C3Fast, the mAP is increased by 0.2%, the calculation amount is reduced by 20%, the parameter amount is reduced by 18%, the weight file size is reduced by 27%, and the detection time is reduced by 0.5ms; after replacing the CBS module with GhostConv, the mAP is improved by another 0.2%, the parameter amount and calculation amount are significantly reduced again, the weight file is further reduced, and the detection time is slightly reduced; finally, after the SE attention mechanism is introduced, the calculation amount remains unchanged, the detection time does not increase, only a small amount of parameters is added, the weight file size is only increased by 0.07MB, and the mAP is further improved; compared with the original model, the parameters of the final model are reduced by 34.5%, the calculation amount is reduced by 34.9%, the weight is reduced by 33.2%, and the detection time is reduced by 0.7ms. Among them, the average precision (mAP) of all classes represents the mean of the area enclosed by the PR curve of each category and the coordinate axis. It is a comprehensive evaluation of the performance of P and R. mAP@0.5 represents the mAP when the intersection over union (IOU) is 0.5.

[0041] In order to more intuitively show the improved model performance, Figure 6 The mAP comparison chart of the original model (the existing YOLOv5 network model) and the improved YOLOv5 model (that is, the YOLOv5-FGS network model in the present invention) during training is shown.

[0042] Figure 7 This is a detection effect diagram of some pictures of the test set on the final improved YOLOv5-FGS network model in an embodiment of the present invention. It can be seen that all relevant gestures and irrelevant hands can be correctly located (regardless of the target size), and the confidence of the relevant gestures that are focused on is above 90%.

[0043] The step S2 includes: using the YOLOv5-FGS hand detection model to classify a gesture dataset containing 11 types of gestures (the dataset example is as follows: Figure 8 ) to detect the hand, locate the positions of relevant gestures and irrelevant gestures, and then use Mediapipe to detect the hand key points of the located relevant gestures, extract and process the features based on the obtained hand key point information, input the final hand key point features and the corresponding gesture labels into the naive Bayes classifier for training, and generate a gesture classification model;

[0044] In step S2, the hand key point feature processing process includes: Fig. 9 As shown, Mediapipe performs hand key point detection to obtain 21 hand key point coordinates, including the wrist point and 4 joint points corresponding to the five fingers. According to the 21 hand key point coordinates, 5 groups of angle features and 5 groups of distance features are extracted as the basis for judging the bending of the fingers, and 4 groups of angle features are extracted as the basis for judging the opening and closing of the fingers. A total of 14 groups of features are obtained, and then the threshold is set to perform binarization processing on the 14 groups of features to obtain the hand key point features;

[0045] like Fig. 9 As shown, the 21 key points of the hand include the wrist point (point 0), the middle point between the wrist and the root of the thumb (point 1), the root of the thumb (point 2), the thumb interphalangeal joint point (point 3), the thumb apex (point 4), the root of the index finger (point 5), the second interphalangeal joint point of the index finger (point 6), the first interphalangeal joint point of the index finger (point 7), the apex of the index finger (point 8), the root of the middle finger (point 9), the second interphalangeal joint point of the middle finger (point 10), the first interphalangeal joint point of the middle finger (point 11), the apex of the middle finger (point 12), the root of the ring finger (point 13), the second interphalangeal joint point of the ring finger (point 14), the first interphalangeal joint point of the ring finger (point 15), the apex of the ring finger (point 16), the root of the little finger (point 17), the second interphalangeal joint point of the little finger (point 18), the first interphalangeal joint point of the little finger (point 19), and the apex of the little finger (point 20). The 5 groups of angle features correspond to the bending angle θ, the 5 groups of distance features correspond to the modulus ratio d, and the 4 groups of angle features correspond to the opening and closing angle a. Their relationship with the 21 hand key points is shown in Table 2:

[0046] Table 214 comparison of the relationship between the features and 21 key points of the hand

[0047]

[0048] Here we take the bending of the thumb as an example to judge the bending of fingers. Fig.10 (a)(b)(c) are schematic diagrams showing the degree of bending of the thumb. Fig.10 (a) shows the effect of the model identifying 21 key points of the hand. Fig.10 (b) is a schematic diagram of the distance ratio for judging the bending condition of the thumb. In the figure, the base of the little finger (point 17) and the base of the thumb (point 2) are vector A (17, 2), and the base of the little finger (point 17) and the top of the thumb (point 4) are vector B (17, 4). The modulus ratio d of the two vectors can reflect the bending condition of the thumb. The calculation method is shown in formula (1):

[0049]

[0050] Among them, x2 and y2 are the coordinates of the base of the thumb (point 2), x 17 ,y 17is the coordinate of the base of the little finger (point 17), x4 and y4 are the coordinates of the top of the thumb (point 4);

[0051] Fig.10 (c) is a schematic diagram of the angle for judging the bending of the thumb. In the figure, the base of the thumb (point 2) and the wrist point (point 0) are vector C(2,0), and the base of the thumb (point 2) and the top of the thumb (point 4) are vector D(2,4). The angle θ between the two vectors can reflect the degree of thumb bending. The calculation formula is shown in formula (2):

[0052]

[0053] Among them, x2 and y2 are the coordinates of the base of the thumb (point 2), x4 and y4 are the coordinates of the top of the thumb (point 4), and x0 and y0 are the coordinates of the wrist point (point 0);

[0054] Here we take the opening and closing of the index finger and the middle finger as an example. Fig.11 (a) (b) are schematic diagrams showing the angles for judging the opening and closing of the index finger and the middle finger. Fig.11 (a) shows the effect of the model identifying 21 key points of the hand. Fig.11 (b) is a schematic diagram of the angle for judging the opening and closing of the index finger and the middle finger. In the figure, the base of the index finger (point 5) and the top of the index finger (point 8) are vector A (5, 8), and the base of the index finger (point 5) and the top of the middle finger (point 12) are vector B (5, 12). The degree of the angle a between the two vectors can reflect the degree of bending of the thumb, and the calculation method is the same as the above formula (2).

[0055] The feature calculation methods for the remaining finger bending conditions and the opening and closing conditions of the fingers are the same as above. After calculation, 14 sets of feature values ​​can be obtained. Since the extracted geometric features are divided into angles and distance ratios, they have different physical meanings, have large value ranges, and are highly complex, here we set a certain threshold and convert the extracted angle features and distance ratio features into binary features.

[0056] 5 groups of angle features correspond to the bending angle θ, 5 groups of distance features correspond to the modulus ratio d, and 4 groups of angle features correspond to the opening and closing angle a. In some embodiments of the present invention, the bending angle θ takes 160 degrees as the threshold, and is converted to 1 if it is less than or equal to, and is converted to 0 if it is greater than. The opening and closing angle a takes 30 degrees as the threshold, and is converted to 1 if it is greater than, and is converted to 0 if it is less than or equal to 30 degrees. The modulus ratio d takes length 1 as the threshold, and is converted to 1 if it is greater than, and is converted to 0 if it is less than or equal to. After the above processing, a 14-dimensional feature vector with eigenvalues ​​of 0 or 1 is obtained.

[0057] The step S3 includes: using the YOLOv5-FGS hand detection model to detect the detected single-frame video image, locate the relevant gestures and irrelevant gesture positions, and then use Mediapipe to detect the hand key points of the located relevant gestures, extract and process the features according to the obtained hand key point information, and input the final hand key point features into the gesture classification model trained in step S2 to obtain the final gesture target positioning result and classification result. The confusion matrix of the final gesture recognition classification result is as follows: Fig.12 shown.

[0058] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned gesture recognition methods based on YOLOv5-FGS and Mediapipe is implemented. The memory may be various types of memory, such as random access memory, read-only memory, flash memory, etc. The processor may be various types of processors, such as a central processing unit, a microprocessor, a digital signal processor, or an image processor, etc.

[0059] The present invention also discloses a computer-readable storage medium storing computer-executable instructions for executing any of the above-mentioned gesture recognition methods based on YOLOv5-FGS and Mediapipe. The storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc., which can store program codes.

[0060] The present invention combines the advantages of deep learning and traditional methods and improves their shortcomings. Firstly, the original YOLOv5 network is lightweighted to obtain YOLOv5-FGS. Without affecting its accuracy, the calculation amount and parameter amount of the model are greatly reduced, the reasoning speed of the network model is accelerated, and the memory occupied by the model is reduced so that it can be better deployed on edge devices with limited resources. Secondly, the YOLOv5-FGS network is creatively used to train a two-classification hand detection model to exclude irrelevant gestures and locate relevant gestures. It can be used as a general hand detection model to increase gesture categories. It only needs to use the general hand detection model to process the data set once, and then use the traditional machine learning classification method (only a small number of samples are needed) to quickly train a new gesture classification model, which effectively solves the problem of high secondary development cost of gesture recognition methods based on deep learning. Finally, the Mediapipe-based hand key point detection method is introduced into the feature extraction stage of the classification model, which greatly improves the classification accuracy.

[0061] In addition, the method of the present invention processes gesture detection and gesture analysis separately, which can improve the performance of the detection model and the classification model respectively. If new gestures need to be added in the later stage, there is no need to train hand detection again. It is only necessary to use the trained YOLOv5-FGS hand detection model to perform hand target detection on the new gesture classification data set to locate the effective gesture position. This can effectively reduce the time cost of training and can be used in different scenarios in gesture recognition tasks, and has a good application prospect.

[0062] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A gesture recognition method based on YOLOv5-FGS and Mediapipe, characterized in that: The following steps are involved: Step S1, constructing a hand detection data set and training an improved YOLOv5-FGS hand detection model; Step S2: Use the YOLOv5-FGS hand detection model to perform hand detection on the gesture classification dataset, perform hand key point detection through Mediapipe, and train the naive Bayes classifier based on the hand key point information output by Mediapipe to generate a gesture classification model. Step S3: For the video image to be detected, first perform single-frame detection through the YOLOv5-FGS hand detection model obtained in step S1 to locate the hand position, and then perform hand key point detection through Mediapipe. The processed hand key point features are detected by the gesture classification model to output the final gesture target positioning result and classification result.

2. The gesture recognition method based on YOLOv5-FGS and Mediapipe according to claim 1, characterized in that: In step S1, the hand detection data set construction process includes: processing an image data set including 18 categories of common gestures and one irrelevant gesture to obtain a binary classification data set containing only relevant gestures and irrelevant gestures, and the ratio of relevant gestures to irrelevant gestures is 3:1; performing data processing on the binary classification data set, adjusting the images in the binary classification data set to images of 640*640 size, and obtaining the final hand detection data set.

3. The gesture recognition method based on YOLOv5-FGS and Mediapipe according to claim 1, characterized in that: In the step S1, the improved YOLOv5-FGS hand detection model structure includes: firstly reconstructing the C3 model in the original YOLOv5 network using the FastNet network to obtain the C3fast model, and replacing the C3 module in the original YOLOv5 network with the C3fast; then replacing the CBS module of the YOLOv5 network (except the CBS module of the first layer) with the Ghost convolution module; and finally adding the SE attention mechanism module to the end of the YOLOv5 feature extraction network.

4. The gesture recognition method based on YOLOv5-FGS and Mediapipe according to claim 1, characterized in that: In the step S2, the hand key point feature processing process includes: Mediapipe performs hand key point detection to obtain 21 hand key point coordinates, including the wrist point and 4 joint points corresponding to the five fingers respectively, and extracts 5 groups of angle features and 5 groups of distance features as the basis for judging the bending of the fingers according to the 21 hand key point coordinates, and extracts 4 groups of angle features as the basis for judging the opening and closing of the fingers, so as to obtain a total of 14 groups of features, and then sets a threshold to perform binarization processing on the 14 groups of features to obtain the hand key point features.

5. The gesture recognition method based on YOLOv5-FGS and Mediapipe according to claim 4, characterized in that: The 5 groups of angle features correspond to the bending angle θ, the 5 groups of distance features correspond to the modulus ratio d, and the 4 groups of angle features correspond to the opening and closing angle a. The bending angle θ takes 160 degrees as the threshold, and is converted to 1 when it is less than or equal to, and is converted to 0 when it is greater than. The opening and closing angle a takes 30 degrees as the threshold, and is converted to 1 when it is greater than, and is converted to 0 when it is less than or equal to 30 degrees. The modulus ratio d takes length 1 as the threshold, and is converted to 1 when it is greater than, and is converted to 0 when it is less than or equal to. Finally, a 14-dimensional feature vector with a feature value of 0 or 1 is obtained as the hand key point feature.

6. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a gesture recognition method based on YOLOv5-FGS and Mediapipe as described in any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute a gesture recognition method based on YOLOv5-FGS and Mediapipe as described in any one of claims 1 to 5.

Images