A gesture recognition method and system based on key points of the human hand
By constructing a database of key points on the human hand and using the nearest neighbor distance threshold to recognize gestures, the problems of large data requirements and high computing resources in existing technologies are solved. This achieves low-cost, high-precision gesture recognition and the ability to quickly expand to new gestures, making it suitable for embedded devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUNAN FIRST NORMAL UNIV
- Filing Date
- 2026-02-04
- Publication Date
- 2026-04-17
AI Technical Summary
Existing gesture recognition technologies require a large amount of training data and high computing resources. Model training is complex and it is difficult to quickly expand to new gestures. Furthermore, the recognition accuracy is not high in complex environments.
By constructing a database of key points on the human hand, calculating the physical parameters between key points to generate key point vectors, and using nearest neighbor distance and adaptive thresholds for recognition, no deep model training is required. When adding new gestures, only the vector library needs to be expanded to adapt to changes in light and angle.
It reduces data requirements and computing resources, lowers deployment costs, supports rapid expansion of new gestures, maintains high recognition accuracy and robustness, and is suitable for embedded devices.
Smart Images
Figure CN121640522B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of gesture recognition technology, specifically to a gesture recognition method and system based on key points of the human hand. Background Technology
[0002] Gesture recognition, as a crucial human-computer interaction technology, has been widely applied in smart homes, autonomous driving, and industrial automation in recent years. Existing gesture recognition technologies are mainly divided into two categories. One category is image classification methods based on deep learning, which train models such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) by collecting a large number of gesture images, and then use these models to classify and recognize input images, such as Google's Inception model and Facebook's ResNet model. The other category is based on traditional feature extraction methods, which extract features such as texture, contour, and color from images and combine them with classifiers such as Support Vector Machines (SVMs) and decision trees to achieve recognition. With the development of computer vision technology, hand keypoint detection technology is becoming increasingly mature.
[0003] Existing technologies can recognize gestures based on key points on the human hand. However, traditional gesture recognition methods have the following shortcomings:
[0004] 1. The demand for various gesture images in the training set is enormous.
[0005] Traditional deep learning-based methods require massive amounts of gesture images as training data (typically thousands of samples for each gesture), and must cover different postures, angles, and environmental conditions. The workload for data collection and annotation is enormous, resulting in high time and manpower costs. For niche gestures or customized scenarios, it is difficult to obtain sufficient training data.
[0006] 2. The model training process is complex.
[0007] Traditional gesture recognition methods not only rely on massive amounts of data for model training but also suffer from complex processes, difficult parameter tuning, and high time and labor costs. Before training, tedious preprocessing such as image normalization and data augmentation is required. During training, the network structure and parameters must be repeatedly optimized, resulting in long training cycles. After training, errors need to be evaluated using a confusion matrix, and classes with high misclassification rates need supplementary data for retraining, often requiring multiple iterations. The entire process demands high levels of expertise from personnel, consumes high-performance computing resources, and severely limits the practical application and widespread adoption of the technology.
[0008] 3. The model needs to be retrained to recognize new types of gestures.
[0009] When adding new gesture types, traditional methods require acquiring a large number of new gesture images and fine-tuning or retraining the existing model. This is not only time-consuming but may also lead to a decrease in the recognition accuracy of existing gestures, making system maintenance difficult and unable to meet the needs of rapidly iterating applications.
[0010] To address the aforementioned issues, there is an urgent need for a gesture recognition method and system based on key points of the human hand to solve the problems existing in traditional methods. Summary of the Invention
[0011] The purpose of this invention is to provide a gesture recognition method and system based on key points of the human hand. By constructing a gesture feature vector library through key point serialization, and combining nearest neighbor distance and adaptive threshold judgment, there is no need to train a deep model, which reduces data requirements and computing resource consumption. When adding a gesture, only the vector library needs to be expanded, which is flexible and convenient. By using the physical features of key points for matching, it maintains high recognition accuracy and robustness under changing lighting and angle environments, and is suitable for embedded devices, etc.
[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0013] A gesture recognition method based on key points of the human hand includes:
[0014] Step 1: Construct a human hand key point database, which includes a database of the same type of gesture data and a database of multiple types of gesture data, specifically including:
[0015] Step 101: Acquire an image containing a human hand;
[0016] Step 102: Detect the coordinates of key points of the human hand in the image;
[0017] Step 103: Calculate the physical quantity parameters between key points based on the key point coordinates, and construct the key point vector based on the physical quantity parameters and key point coordinates;
[0018] Step 104: Obtain key point vectors from multiple images of the same gesture and construct a database of data for the same gesture;
[0019] Step 105: Merge all the gesture data databases to obtain the final database of multiple gesture data;
[0020] Step 2: For a certain gesture, calculate the intra-class nearest neighbor distance and inter-class nearest neighbor distance for that gesture, and determine the nearest neighbor distance threshold based on the intra-class nearest neighbor distance and inter-class nearest neighbor distance;
[0021] Step 3: Acquire the image to be recognized and recognize the gesture based on the nearest neighbor distance threshold.
[0022] Furthermore, in step 102, the key points of the human hand in the image include points on the palm and points on the finger joints.
[0023] Furthermore, in step 103, the physical quantity parameters include the angle and distance between key points.
[0024] Further, in step 2, for a given gesture, the intra-class nearest neighbor distance for that gesture is calculated, specifically as follows:
[0025] For a given gesture, calculate the nearest neighbor distance among all keypoint vectors of that gesture, i.e., the intra-class nearest neighbor distance, where each keypoint vector of that gesture has an intra-class nearest neighbor distance.
[0026] Further, in step 2, for a given gesture, the inter-class nearest neighbor distance for that gesture is calculated, specifically as follows:
[0027] For a given gesture, calculate the nearest neighbor distance between all keypoint vectors of that gesture and all keypoint vectors of all other gestures, i.e., the inter-class nearest neighbor distance.
[0028] Further, in step 2, the nearest neighbor distance threshold is determined based on the intra-class nearest neighbor distance and the inter-class nearest neighbor distance, specifically as follows:
[0029] For a certain gesture, assign the label 1 to the intra-class nearest neighbor distance of the gesture, and assign the label 0 to the inter-class nearest neighbor distance of the gesture;
[0030] For a certain gesture, the intra-class nearest neighbor distance of each key point vector of the gesture is used as the threshold. The intra-class nearest neighbor distance and inter-class nearest neighbor distance of the gesture are then divided into binary categories. Those not greater than the threshold are predicted as 1, and those greater than the threshold are predicted as 0. The F1 score is then calculated.
[0031] For a certain gesture, the intra-class nearest neighbor distance corresponding to the maximum F1 value is determined as the threshold of that gesture. For multiple gestures, the threshold of each gesture is calculated in turn according to the same process to obtain the threshold of each gesture.
[0032] Further, in step 3, the image to be recognized is acquired, and the gesture is recognized based on the nearest neighbor distance threshold, specifically as follows:
[0033] Acquire the image to be recognized;
[0034] Perform keypoint serialization on the image to be recognized;
[0035] Calculate the nearest neighbor distance based on key point vectors;
[0036] The gesture type is determined based on the nearest neighbor distance and the gesture threshold.
[0037] Furthermore, the method also includes: adding new gestures, specifically:
[0038] Build a new gesture database and add it to the human hand key point database;
[0039] Calculate the nearest neighbor distance for the new gesture and the gesture threshold.
[0040] This invention also provides a gesture recognition system based on key points of the human hand, applied to the above-mentioned gesture recognition method based on key points of the human hand, comprising:
[0041] The image acquisition module is used to acquire images containing human hands;
[0042] The key point detection module, connected to the image acquisition module, is used to detect the coordinates of key points of the human hand in the image;
[0043] The vector construction module connects to the keypoint detection module and is used to calculate the angle and distance between keypoints based on their coordinates, and then generate keypoint vectors by combining the coordinates.
[0044] The database module and the connection vector construction module are used to store sub-databases composed of multiple key point vectors of the same gesture and the total database formed by the fusion of multiple gesture sub-databases.
[0045] The distance calculation module connects the vector construction module and the database module, and is used to calculate the nearest neighbor distance between the vector of the key point to be identified and each vector in the database.
[0046] The threshold judgment module, connected to the distance calculation module, is used to determine the type of gesture based on the nearest neighbor distance and a preset threshold.
[0047] The output module, connected to the threshold judgment module, is used to output the recognition results.
[0048] In summary, the present invention has at least one of the following beneficial technical effects:
[0049] 1. Low data demand and low deployment cost
[0050] Traditional gesture recognition methods require collecting thousands or even tens of thousands of gesture images for model training, while this invention only requires a few or dozens of images of each gesture to build an effective database. This significantly reduces the workload of data collection and substantially lowers the cost of data collection and storage.
[0051] 2. No model training process, low resource consumption
[0052] Traditional methods require significant computational resources for model training (typically requiring several hours of operation on a high-performance GPU), and the trained models are large in size, making them difficult to deploy on embedded devices. This invention eliminates the need for model training, has low computational resource requirements, and can run smoothly on resource-constrained devices.
[0053] 3. The new gestures are easy to expand and highly flexible.
[0054] Traditional methods require acquiring a large number of new images and retraining the entire model when adding new gestures, which may affect the recognition accuracy of existing gestures. This invention, however, only requires acquiring a small number of images, supplementing the database, and calculating thresholds when adding new gestures. No modifications to existing system settings are needed, and the recognition performance of existing gestures remains unaffected.
[0055] 4. High recognition accuracy and strong environmental adaptability
[0056] In complex environments (such as changes in lighting, shifts in shooting angle, and slight obstruction), the gesture recognition accuracy of this invention is higher. Attached Figure Description
[0057] Figure 1 This is a schematic flowchart of the method of the present invention;
[0058] Figure 2 This is a hardware architecture diagram of the system of the present invention. Detailed Implementation
[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0060] like Figure 1 As shown, the present invention provides a gesture recognition method based on key points of the human hand, including:
[0061] Step 1: Construct a human hand key point database, which includes a database of the same type of gesture data and a database of multiple types of gesture data, specifically including:
[0062] Step 101: Acquire an image containing a human hand;
[0063] Step 102: Detect the coordinates of key points of the human hand in the image. This invention can use OpenPose or Mediapipe as the key point detection model.
[0064] Step 103: Calculate the physical quantity parameters between key points based on the key point coordinates, and construct the key point vector based on the physical quantity parameters and key point coordinates;
[0065] Step 104: Collect multiple images of the same gesture. During the collection process, while keeping the meaning of the gesture unchanged, adjust the position, angle and expression of the gesture. Serialize the key points of the gesture in each image to obtain high-dimensional vectors of the gesture in all images. These high-dimensional vectors together constitute a database of this gesture.
[0066] Step 105: Construction of multiple gesture databases. Following the method in step 104, construct databases for multiple gestures. Each gesture database has its own unique number. Combine all the gesture databases to form the final multiple gesture database.
[0067] Step 2: For a certain gesture, calculate the intra-class nearest neighbor distance and inter-class nearest neighbor distance for that gesture, and determine the nearest neighbor distance threshold based on the intra-class nearest neighbor distance and inter-class nearest neighbor distance;
[0068] Step 3: Acquire the image to be recognized and recognize the gesture based on the nearest neighbor distance threshold.
[0069] The present invention also provides an embodiment in which Manhattan distance is used instead of Euclidean distance to calculate vector similarity.
[0070] In step 102, the key points of the human hand in the image include points on the palm and points on the finger joints.
[0071] In step 103, the physical quantity parameters between key points are calculated based on the key point coordinates, and a key point vector is constructed based on the physical quantity parameters and the key point coordinates, specifically as follows:
[0072] Based on the coordinates of keypoints, physical quantities such as angles and distances between keypoints are calculated. These physical quantities, together with the coordinates of the keypoints, form a high-dimensional vector, thus realizing keypoint serialization. The mathematical expression of this process is as follows:
[0073] (1)
[0074] (2)
[0075] (3)
[0076] In the formula, Indicates the angle between key points. Indicates the distance between key points. These represent the coordinates of the key points. Combining these three quantities into a single vector yields:
[0077] (4)
[0078] In the formula, This is the high-dimensional vector obtained through serialization.
[0079] In step 2, for a given gesture, the intra-class nearest neighbor distance for that gesture is calculated as follows:
[0080] For a given gesture, calculate the nearest neighbor distance among all keypoint vectors of that gesture, i.e., the intra-class nearest neighbor distance, where each keypoint vector of that gesture has an intra-class nearest neighbor distance.
[0081] In step 2, for a given gesture, the inter-class nearest neighbor distance for that gesture is calculated, specifically as follows:
[0082] For a given gesture, calculate the nearest neighbor distance between all keypoint vectors of that gesture and all keypoint vectors of all other gestures, i.e., the inter-class nearest neighbor distance.
[0083] In step 2, the nearest neighbor distance threshold is determined based on the intra-class nearest neighbor distance and the inter-class nearest neighbor distance, specifically as follows:
[0084] For a certain gesture, assign the label 1 to the intra-class nearest neighbor distance of the gesture, and assign the label 0 to the inter-class nearest neighbor distance of the gesture;
[0085] For a certain gesture, the intra-class nearest neighbor distance of each key point vector of the gesture is used as the threshold. The intra-class nearest neighbor distance and inter-class nearest neighbor distance of the gesture are then divided into binary categories. Those not greater than the threshold are predicted as 1, and those greater than the threshold are predicted as 0. The F1 score is then calculated.
[0086] For a given gesture, the intra-class nearest neighbor distance corresponding to the maximum F1 value is determined as the threshold T for that gesture. For multiple gestures, the threshold for each gesture is calculated sequentially using the same process, resulting in the thresholds {T1, T2, ..., T} for each gesture. i}
[0087] The present invention can also use the optimal threshold point of the ROC curve (Receptor Operating Characteristic curve) instead of the F1 value maximization method. By calculating the Youden index (sensitivity + specificity - 1) of the ROC curve, the distance corresponding to the maximum Youden index is selected as the threshold.
[0088] In step 3, the image to be recognized is acquired, and the gesture is recognized based on the nearest neighbor distance threshold, specifically as follows:
[0089] Image acquisition is performed using an image acquisition module.
[0090] Keypoint serialization involves detecting the coordinates of key points on the human hand in the image and serializing these coordinates in the same way as building a database of key points on the human hand.
[0091] Calculate the nearest neighbor distance: find the vector that is closest to the hand keypoint in the database, and calculate the distance d between the hand keypoint vector and its nearest neighbor vector.
[0092] The gesture types are initially determined by identifying the gesture type corresponding to the nearest neighbor vector.
[0093] Finally, the gesture type is determined by comparing the nearest neighbor distance d with the threshold T of the corresponding gesture type. If d is not greater than T, the gesture in the image is identified as that gesture; if d is greater than T, the gesture in the image is identified as an unknown gesture.
[0094] The method further includes: adding new gestures, specifically:
[0095] Build a new gesture database and add it to the human hand key point database;
[0096] Calculate the nearest neighbor distance and the threshold of the new gesture;
[0097] Recognize new gestures.
[0098] It should also be noted that the present invention can use a vector database to replace the traditional storage unit to store high-dimensional vectors, and the vector database supports fast similarity retrieval algorithms.
[0099] This invention also provides a gesture recognition system based on key points of the human hand, applied to the above-mentioned gesture recognition method based on key points of the human hand, comprising:
[0100] The image acquisition module is used to acquire images containing human hands;
[0101] The key point detection module, connected to the image acquisition module, is used to detect the coordinates of key points of the human hand in the image;
[0102] The vector construction module connects to the keypoint detection module and is used to calculate the angle and distance between keypoints based on their coordinates, and then generate keypoint vectors by combining the coordinates.
[0103] The database module and the connection vector construction module are used to store sub-databases composed of multiple key point vectors of the same gesture and the total database formed by the fusion of multiple gesture sub-databases.
[0104] The distance calculation module connects the vector construction module and the database module, and is used to calculate the nearest neighbor distance between the vector of the key point to be identified and each vector in the database.
[0105] The threshold judgment module, connected to the distance calculation module, is used to determine the type of gesture based on the nearest neighbor distance and a preset threshold.
[0106] The output module, connected to the threshold judgment module, is used to output the recognition results.
[0107] like Figure 2 As shown, the present invention also provides a hardware system architecture diagram, wherein the image acquisition module can be a camera, the output module can be a display screen, and the remaining modules are all built inside the computing device.
[0108] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0109] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.
[0112] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.
Claims
1. A method for gesture recognition based on human hand key points, characterized in that, include: Step 1: Construct a human hand key point database, which includes a database of the same type of gesture data and a database of multiple types of gesture data, specifically including: Step 101: Acquire an image containing a human hand; Step 102: Detect the coordinates of key points of the human hand in the image; Step 103: Calculate the physical quantity parameters between key points based on the key point coordinates, and construct the key point vector based on the physical quantity parameters and key point coordinates; Step 104: Obtain key point vectors from multiple images of the same gesture and construct a database of data for the same gesture; Step 105: Merge all the gesture data databases to obtain the final database of multiple gesture data; Step 2: For a given gesture, calculate the intra-class nearest neighbor distance and the inter-class nearest neighbor distance for that gesture, and determine the nearest neighbor distance threshold based on the intra-class nearest neighbor distance and the inter-class nearest neighbor distance; specifically, determining the nearest neighbor distance threshold based on the intra-class nearest neighbor distance and the inter-class nearest neighbor distance involves: For a certain gesture, assign the label 1 to the intra-class nearest neighbor distance of the gesture, and assign the label 0 to the inter-class nearest neighbor distance of the gesture; For a certain gesture, the intra-class nearest neighbor distance of each key point vector of the gesture is used as the threshold. The intra-class nearest neighbor distance and inter-class nearest neighbor distance of the gesture are divided into binary values. Those not greater than the threshold are predicted as 1, and those greater than the threshold are predicted as 0. The F1 value is calculated. For a certain gesture, the intra-class nearest neighbor distance corresponding to the maximum F1 value is determined as the threshold of that gesture. For multiple gestures, the threshold of each gesture is calculated in turn according to the same process to obtain the threshold of each gesture. Step 3: Acquire the image to be recognized and recognize the gesture based on the nearest neighbor distance threshold.
2. The gesture recognition method based on key points of the human hand according to claim 1, characterized in that, In step 102, the key points of the human hand in the image include points on the palm and points on the finger joints.
3. The gesture recognition method based on key points of the human hand according to claim 2, characterized in that, In step 103, the physical quantity parameters include the angle and distance between key points.
4. The gesture recognition method based on key points of the human hand according to claim 3, characterized in that, In step 2, for a given gesture, the intra-class nearest neighbor distance for that gesture is calculated as follows: For a given gesture, calculate the nearest neighbor distance among all keypoint vectors of that gesture, i.e., the intra-class nearest neighbor distance, where each keypoint vector of that gesture has an intra-class nearest neighbor distance.
5. The gesture recognition method based on key points of the human hand according to claim 4, characterized in that, In step 2, for a given gesture, the inter-class nearest neighbor distance for that gesture is calculated, specifically as follows: For a given gesture, calculate the nearest neighbor distance between all keypoint vectors of that gesture and all keypoint vectors of all other gestures, i.e., the inter-class nearest neighbor distance.
6. The gesture recognition method based on key points of the human hand according to claim 5, characterized in that, In step 3, the image to be recognized is acquired, and the gesture is recognized based on the nearest neighbor distance threshold, specifically as follows: Acquire the image to be recognized; Perform keypoint serialization on the image to be recognized; Calculate the nearest neighbor distance based on key point vectors; The gesture type is determined based on the nearest neighbor distance and the gesture threshold.
7. The gesture recognition method based on key points of the human hand according to claim 6, characterized in that, The method further includes: adding new gestures, specifically: Build a new gesture database and add it to the human hand key point database; Calculate the nearest neighbor distance for the new gesture and the gesture threshold.
8. A gesture recognition system based on key points of the human hand, applied to the gesture recognition method based on key points of the human hand as described in any one of claims 1-7, characterized in that, include: The image acquisition module is used to acquire images containing human hands; The key point detection module, connected to the image acquisition module, is used to detect the coordinates of key points of the human hand in the image; The vector construction module connects to the keypoint detection module and is used to calculate the angle and distance between keypoints based on their coordinates, and then generate keypoint vectors by combining the coordinates. The database module and the connection vector construction module are used to store sub-databases composed of multiple key point vectors of the same gesture and the total database formed by the fusion of multiple gesture sub-databases. The distance calculation module connects the vector construction module and the database module, and is used to calculate the nearest neighbor distance between the vector of the key point to be identified and each vector in the database; the nearest neighbor distance threshold is determined based on the intra-class nearest neighbor distance and the inter-class nearest neighbor distance, specifically: For a certain gesture, assign the label 1 to the intra-class nearest neighbor distance of the gesture, and assign the label 0 to the inter-class nearest neighbor distance of the gesture; For a certain gesture, the intra-class nearest neighbor distance of each key point vector of the gesture is used as the threshold. The intra-class nearest neighbor distance and inter-class nearest neighbor distance of the gesture are divided into binary values. Those not greater than the threshold are predicted as 1, and those greater than the threshold are predicted as 0. The F1 value is calculated. For a certain gesture, the intra-class nearest neighbor distance corresponding to the maximum F1 value is determined as the threshold of that gesture. For multiple gestures, the threshold of each gesture is calculated in turn according to the same process to obtain the threshold of each gesture. The threshold judgment module, connected to the distance calculation module, is used to determine the type of gesture based on the nearest neighbor distance and a preset threshold. The output module, connected to the threshold judgment module, is used to output the recognition results.