Gesture recognition method and system based on hand key points

Abstract

The invention discloses a gesture recognition method and system based on hand key points, and belongs to the technical field of gesture recognition. The method comprises the steps that a human hand key point database is constructed, the human hand key point database comprises a database of the same gesture data and a database of various gesture data, and for a certain gesture, the intra-class nearest neighbor distance and the inter-class nearest neighbor distance of the gesture are calculated; and determining a nearest neighbor distance threshold value based on the intra-class nearest neighbor distance and the inter-class nearest neighbor distance, obtaining a to-be-recognized image, and recognizing a gesture based on the nearest neighbor distance threshold value. According to the method, the gesture feature vector library is constructed through key point serialization, the nearest neighbor distance and adaptive threshold judgment are combined, a depth model does not need to be trained, and data requirements and computing resource occupation are reduced; only the vector library needs to be expanded when gestures are newly added, and expansion is flexible and convenient; matching is carried out by utilizing physical characteristics of key points, high identification precision and robustness are still kept in a light and angle changing environment, and the method is suitable for embedded equipment and the like.

Description

Technical Field

[0001] This invention relates to the field of gesture recognition technology, specifically to a gesture recognition method and system based on key points of the human hand. Background Technology

[0002] Gesture recognition, as a crucial human-computer interaction technology, has been widely applied in smart homes, autonomous driving, and industrial automation in recent years. Existing gesture recognition technologies are mainly divided into two categories. One category is image classification methods based on deep learning, which train models such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) by collecting a large number of gesture images, and then use these models to classify and recognize input images, such as Google's Inception model and Facebook's ResNet model. The other category is based on traditional feature extraction methods, which extract features such as texture, contour, and color from images and combine them with classifiers such as Support Vector Machines (SVMs) and decision trees to achieve recognition. With the development of computer vision technology, hand keypoint detection technology is becoming increasingly mature.

[0003] Existing technologies can recognize gestures based on key points on the human hand. However, traditional gesture recognition methods have the following shortcomings: 1. The demand for various gesture images in the training set is enormous. Traditional deep learning-based methods require massive amounts of gesture images as training data (typically thousands of samples for each gesture), and must cover different postures, angles, and environmental conditions. The workload for data collection and annotation is enormous, resulting in high time and manpower costs. For niche gestures or customized scenarios, it is difficult to obtain sufficient training data.

[0004] 2. The model training process is complex. Traditional gesture recognition methods not only rely on massive amounts of data for model training but also suffer from complex processes, difficult parameter tuning, and high time and labor costs. Before training, tedious preprocessing such as image normalization and data augmentation is required. During training, the network structure and parameters must be repeatedly optimized, resulting in long training cycles. After training, errors need to be evaluated using a confusion matrix, and classes with high misclassification rates need supplementary data for retraining, often requiring multiple iterations. The entire process demands high levels of expertise from personnel, consumes high-performance computing resources, and severely limits the practical application and widespread adoption of the technology.

[0005] 3. The model needs to be retrained to recognize new types of gestures. When adding new gesture types, traditional methods require acquiring a large number of new gesture images and fine-tuning or retraining the existing model. This is not only time-consuming but may also lead to a decrease in the recognition accuracy of existing gestures, making system maintenance difficult and unable to meet the needs of rapidly iterating applications.

[0006] To address the aforementioned issues, there is an urgent need for a gesture recognition method and system based on key points of the human hand to solve the problems existing in traditional methods. Summary of the Invention

[0007] The purpose of this invention is to provide a gesture recognition method and system based on key points of the human hand. By constructing a gesture feature vector library through key point serialization, and combining nearest neighbor distance and adaptive threshold judgment, there is no need to train a deep model, which reduces data requirements and computing resource consumption. When adding a gesture, only the vector library needs to be expanded, which is flexible and convenient. By using the physical features of key points for matching, it maintains high recognition accuracy and robustness under changing lighting and angle environments, and is suitable for embedded devices, etc.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A gesture recognition method based on key points of the human hand includes: Step 1: Construct a human hand key point database, which includes a database of the same type of gesture data and a database of multiple types of gesture data, specifically including: Step 101: Acquire an image containing a human hand; Step 102: Detect the coordinates of key points of the human hand in the image; Step 103: Calculate the physical quantity parameters between key points based on the key point coordinates, and construct the key point vector based on the physical quantity parameters and key point coordinates; Step 104: Obtain key point vectors from multiple images of the same gesture and construct a database of data for the same gesture; Step 105: Merge all the gesture data databases to obtain the final database of multiple gesture data; Step 2: For a certain gesture, calculate the intra-class nearest neighbor distance and inter-class nearest neighbor distance for that gesture, and determine the nearest neighbor distance threshold based on the intra-class nearest neighbor distance and inter-class nearest neighbor distance; Step 3: Acquire the image to be recognized and recognize the gesture based on the nearest neighbor distance threshold.

[0009] Furthermore, in step 102, the key points of the human hand in the image include points on the palm and points on the finger joints.

[0010] Furthermore, in step 103, the physical quantity parameters include the angle and distance between key points.

[0011] Further, in step 2, for a given gesture, the intra-class nearest neighbor distance for that gesture is calculated, specifically as follows: For a given gesture, calculate the nearest neighbor distance among all keypoint vectors of that gesture, i.e., the intra-class nearest neighbor distance, where each keypoint vector of that gesture has an intra-class nearest neighbor distance.

[0012] Further, in step 2, for a given gesture, the inter-class nearest neighbor distance for that gesture is calculated, specifically as follows: For a given gesture, calculate the nearest neighbor distance between all keypoint vectors of that gesture and all keypoint vectors of all other gestures, i.e., the inter-class nearest neighbor distance.

[0013] Further, in step 2, the nearest neighbor distance threshold is determined based on the intra-class nearest neighbor distance and the inter-class nearest neighbor distance, specifically as follows: For a certain gesture, assign the label 1 to the intra-class nearest neighbor distance of the gesture, and assign the label 0 to the inter-class nearest neighbor distance of the gesture; For a certain gesture, the intra-class nearest neighbor distance of each key point vector of the gesture is used as the threshold. The intra-class nearest neighbor distance and inter-class nearest neighbor distance of the gesture are divided into binary values. Those not greater than the threshold are predicted as 1, and those greater than the threshold are predicted as 0. The F1 value is calculated. For a certain gesture, the intra-class nearest neighbor distance corresponding to the maximum F1 value is determined as the threshold of that gesture. For multiple gestures, the threshold of each gesture is calculated in turn according to the same process to obtain the threshold of each gesture.

[0014] Further, in step 3, the image to be recognized is acquired, and the gesture is recognized based on the nearest neighbor distance threshold, specifically as follows: Acquire the image to be recognized; Perform keypoint serialization on the image to be recognized; Calculate the nearest neighbor distance based on key point vectors; The gesture type is determined based on the nearest neighbor distance and the gesture threshold.

[0015] Furthermore, the method also includes: adding new gestures, specifically: Build a new gesture database and add it to the human hand key point database; Calculate the nearest neighbor distance for the new gesture and the gesture threshold.

[0016] This invention also provides a gesture recognition system based on key points of the human hand, applied to the above-mentioned gesture recognition method based on key points of the human hand, comprising: The image acquisition module is used to acquire images containing human hands; The key point detection module, connected to the image acquisition module, is used to detect the coordinates of key points of the human hand in the image; The vector construction module connects to the keypoint detection module and is used to calculate the angle and distance between keypoints based on their coordinates, and then generate keypoint vectors by combining the coordinates. The database module and the connection vector construction module are used to store sub-databases composed of multiple key point vectors of the same gesture and the total database formed by the fusion of multiple gesture sub-databases. The distance calculation module connects the vector construction module and the database module, and is used to calculate the nearest neighbor distance between the vector of the key point to be identified and each vector in the database. The threshold judgment module, connected to the distance calculation module, is used to determine the type of gesture based on the nearest neighbor distance and a preset threshold. The output module, connected to the threshold judgment module, is used to output the recognition results.

[0017] In summary, the present invention has at least one of the following beneficial technical effects: 1. Low data demand and low deployment cost Traditional gesture recognition methods require collecting thousands or even tens of thousands of gesture images for model training, while this invention only requires a few or dozens of images of each gesture to build an effective database. This significantly reduces the workload of data collection and substantially lowers the cost of data collection and storage.

[0018] 2. No model training process, low resource consumption Traditional methods require significant computational resources for model training (typically requiring several hours of operation on a high-performance GPU), and the trained models are large in size, making them difficult to deploy on embedded devices. This invention eliminates the need for model training, has low computational resource requirements, and can run smoothly on resource-constrained devices.

[0019] 3. The new gestures are easy to expand and highly flexible. Traditional methods require acquiring a large number of new images and retraining the entire model when adding new gestures, which may affect the recognition accuracy of existing gestures. This invention, however, only requires acquiring a small number of images, supplementing the database, and calculating thresholds when adding new gestures. No modifications to existing system settings are needed, and the recognition performance of existing gestures is not affected.

[0020] 4. High recognition accuracy and strong environmental adaptability In complex environments (such as changes in lighting, shifts in shooting angle, and slight obstruction), the gesture recognition accuracy of this invention is higher. Attached Figure Description

[0021] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a hardware architecture diagram of the system of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] like Figure 1 As shown, the present invention provides a gesture recognition method based on key points of the human hand, including: Step 1: Construct a human hand key point database, which includes a database of the same type of gesture data and a database of multiple types of gesture data, specifically including: Step 101: Acquire an image containing a human hand; Step 102: Detect the coordinates of key points of the human hand in the image. This invention can use OpenPose or Mediapipe as the key point detection model. Step 103: Calculate the physical quantity parameters between key points based on the key point coordinates, and construct the key point vector based on the physical quantity parameters and key point coordinates; Step 104: Collect multiple images of the same gesture. During the collection process, while keeping the meaning of the gesture unchanged, adjust the position, angle and expression of the gesture. Serialize the key points of the gesture in each image to obtain high-dimensional vectors of the gestures in all images. These high-dimensional vectors together constitute a database of this gesture. Step 105: Construction of multiple gesture databases. Following the method in step 104, construct databases for multiple gestures. Each gesture database has its own unique number. Combine all the gesture databases to form the final multiple gesture database. Step 2: For a certain gesture, calculate the intra-class nearest neighbor distance and inter-class nearest neighbor distance for that gesture, and determine the nearest neighbor distance threshold based on the intra-class nearest neighbor distance and inter-class nearest neighbor distance; Step 3: Acquire the image to be recognized and recognize the gesture based on the nearest neighbor distance threshold.

[0024] The present invention also provides an embodiment in which Manhattan distance is used instead of Euclidean distance to calculate vector similarity.

[0025] In step 102, the key points of the human hand in the image include points on the palm and points on the finger joints.

[0026] In step 103, the physical quantity parameters between key points are calculated based on the key point coordinates, and a key point vector is constructed based on the physical quantity parameters and the key point coordinates, specifically as follows: Based on the coordinates of keypoints, physical quantities such as angles and distances between keypoints are calculated. These physical quantities, together with the coordinates of the keypoints, form a high-dimensional vector, thus realizing keypoint serialization. The mathematical expression of this process is as follows: (1) (2) (3) In the formula, Indicates the angle between key points. Indicates the distance between key points. These represent the coordinates of the key points. Combining these three quantities into a single vector yields: (4) In the formula, This is the high-dimensional vector obtained through serialization.

[0027] In step 2, for a given gesture, the intra-class nearest neighbor distance for that gesture is calculated as follows: For a given gesture, calculate the nearest neighbor distance among all keypoint vectors of that gesture, i.e., the intra-class nearest neighbor distance, where each keypoint vector of that gesture has an intra-class nearest neighbor distance.

[0028] In step 2, for a given gesture, the inter-class nearest neighbor distance for that gesture is calculated, specifically as follows: For a given gesture, calculate the nearest neighbor distance between all keypoint vectors of that gesture and all keypoint vectors of all other gestures, i.e., the inter-class nearest neighbor distance.

[0029] In step 2, the nearest neighbor distance threshold is determined based on the intra-class nearest neighbor distance and the inter-class nearest neighbor distance, specifically as follows: For a certain gesture, assign the label 1 to the intra-class nearest neighbor distance of the gesture, and assign the label 0 to the inter-class nearest neighbor distance of the gesture; For a certain gesture, the intra-class nearest neighbor distance of each key point vector of the gesture is used as the threshold. The intra-class nearest neighbor distance and inter-class nearest neighbor distance of the gesture are divided into binary values. Those not greater than the threshold are predicted as 1, and those greater than the threshold are predicted as 0. The F1 value is calculated. For a given gesture, the intra-class nearest neighbor distance corresponding to the maximum F1 value is determined as the threshold T for that gesture. For multiple gestures, the threshold for each gesture is calculated sequentially using the same process, resulting in the thresholds {T1, T2, ..., T} for each gesture. i}

[0030] The present invention can also use the optimal threshold point of the ROC curve (Receiving Operator Characteristic curve) instead of the F1 value maximization method. By calculating the Youden index (sensitivity + specificity - 1) of the ROC curve, the distance corresponding to the maximum Youden index is selected as the threshold.

[0031] In step 3, the image to be recognized is acquired, and the gesture is recognized based on the nearest neighbor distance threshold, specifically as follows: Image acquisition is performed using an image acquisition module. Keypoint serialization involves detecting the coordinates of key points on the human hand in the image and serializing these coordinates in the same way as building a database of key points on the human hand. Calculate the nearest neighbor distance: find the vector that is closest to the hand keypoint in the database, and calculate the distance d between the hand keypoint vector and its nearest neighbor vector. The gesture types are initially determined by identifying the gesture type corresponding to the nearest neighbor vector. Finally, the gesture type is determined by comparing the nearest neighbor distance d with the threshold T of the corresponding gesture type. If d is not greater than T, the gesture in the image is identified as that gesture; if d is greater than T, the gesture in the image is identified as an unknown gesture.

[0032] The method further includes: adding new gestures, specifically: Build a new gesture database and add it to the human hand key point database; Calculate the nearest neighbor distance and the threshold of the new gesture; Recognize new gestures.

[0033] It should also be noted that the present invention can use a vector database to replace the traditional storage unit to store high-dimensional vectors, and the vector database supports fast similarity retrieval algorithms.

[0034] This invention also provides a gesture recognition system based on key points of the human hand, applied to the above-mentioned gesture recognition method based on key points of the human hand, comprising: The image acquisition module is used to acquire images containing human hands; The key point detection module, connected to the image acquisition module, is used to detect the coordinates of key points of the human hand in the image; The vector construction module connects to the keypoint detection module and is used to calculate the angle and distance between keypoints based on their coordinates, and then generate keypoint vectors by combining the coordinates. The database module and the connection vector construction module are used to store sub-databases composed of multiple key point vectors of the same gesture and the total database formed by the fusion of multiple gesture sub-databases. The distance calculation module connects the vector construction module and the database module, and is used to calculate the nearest neighbor distance between the vector of the key point to be identified and each vector in the database. The threshold judgment module, connected to the distance calculation module, is used to determine the type of gesture based on the nearest neighbor distance and a preset threshold. The output module, connected to the threshold judgment module, is used to output the recognition results.

[0035] like Figure 2 As shown, the present invention also provides a hardware system architecture diagram, wherein the image acquisition module can be a camera, the output module can be a display screen, and the remaining modules are all built inside the computing device.

[0036] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0037] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0038] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0039] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0040] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A method for gesture recognition based on human hand key points, characterized in that, The method comprises the following steps: Step 1: constructing a human hand key point database, the human hand key point database comprising a database of same gesture data and a database of multiple gesture data, and specifically comprising the following steps: Step 101: collecting an image containing a human hand; Step 102: detecting the key point coordinates of the human hand on the image; Step 103: calculating the physical quantity parameters between the key points based on the key point coordinates, and constructing a key point vector based on the physical quantity parameters and the key point coordinates; Step 104: obtaining the key point vectors calculated from multiple images of the same gesture, and constructing a database of the same gesture data; Step 105: fusing the databases of all gesture data to obtain a final database of multiple gesture data; Step 2: for a certain gesture, calculating the intra-class nearest distance and the inter-class nearest distance of the gesture, and determining the nearest distance threshold based on the intra-class nearest distance and the inter-class nearest distance; Step 3: obtaining a to-be-recognized image, and recognizing the gesture based on the nearest distance threshold.

2. The hand gesture recognition method based on hand key points according to claim 1, characterized in that, In step 102, the key points of the human hand on the image include points on the palm and points on the joints of the fingers.

3. The hand gesture recognition method based on hand key points according to claim 2, characterized in that, In step 103, the physical quantity parameters include the angles and distances between the key points.

4. The hand gesture recognition method based on hand key points according to claim 3, characterized in that, In step 2, for a certain gesture, the intra-class nearest distance of the gesture is calculated, specifically as follows: For a certain gesture, the nearest distance between all key point vectors of the gesture, that is, the intra-class nearest distance, is calculated, wherein each key point vector of the gesture has an intra-class nearest distance.

5. The hand gesture recognition method based on hand key points according to claim 4, characterized in that, In step 2, for a certain gesture, the inter-class nearest distance of the gesture is calculated, specifically as follows: For a certain gesture, the nearest distance between all key point vectors of the gesture and all key point vectors of other gestures, that is, the inter-class nearest distance, is calculated.

6. The hand gesture recognition method based on hand key points according to claim 5, characterized in that, In step 2, the nearest distance threshold is determined based on the intra-class nearest distance and the inter-class nearest distance, specifically as follows: For a certain gesture, the intra-class nearest distance of the gesture is assigned to label 1, and the inter-class nearest distance of the gesture is assigned to label 0; For a certain gesture, the intra-class nearest distance and the inter-class nearest distance of the gesture are binarized by taking the intra-class nearest distance of each key point vector of the gesture as a threshold in turn, and the values less than the threshold are predicted as 1, and the values greater than the threshold are predicted as 0, and the F1 value is calculated; For a certain gesture, the intra-class nearest distance corresponding to the maximum F1 value is determined as the threshold of the gesture, and for multiple gestures, the threshold of each gesture is calculated in the same process to obtain the thresholds of the gestures.

7. The hand gesture recognition method based on hand key points according to claim 6, characterized in that, In step 3, the to-be-recognized image is obtained, and the gesture is recognized based on the nearest distance threshold, specifically as follows: Collecting a to-be-recognized image; Serializing the key points of the to-be-recognized image; Calculating the nearest distance based on the key point vector; Determining the gesture type based on the nearest distance and the threshold of the gesture.

8. The hand gesture recognition method based on hand key points according to claim 7, characterized in that, The method further comprises adding a new gesture, specifically as follows: Constructing a database of the new gesture, and adding the database of the new gesture to the human hand key point database; Calculating the nearest distance of the new gesture and the threshold of the gesture.

9. A hand keypoint-based gesture recognition system, applied to the hand keypoint-based gesture recognition method of any one of claims 1-8, characterized in that, The method comprises the following steps: An image collection module is configured to collect an image containing a human hand; A key point detection module is connected with the image acquisition module and is configured to detect hand key point coordinates in the image. A vector construction module is connected with the key point detection module and is configured to calculate angles and distances between the key points according to the key point coordinates and to generate key point vectors in combination with the coordinates. A database module is connected with the vector construction module and is configured to store a sub-database composed of multiple key point vectors of the same gesture and a total database formed by fusing multiple gesture sub-databases. A distance calculation module is connected with the vector construction module and the database module and is configured to calculate the nearest distance between a to-be-identified key point vector and each vector in the database. A threshold judgment module is connected with the distance calculation module and is configured to judge the gesture type based on the nearest distance and a preset threshold. An output module is connected with the threshold judgment module and is configured to output a recognition result.

Images