Gesture recognition method and device, electronic equipment and storage medium

Abstract

The invention discloses a gesture recognition method and device and electronic equipment, and belongs to the technical field of gesture recognition. The method comprises the following steps: carrying out hand detection on a picture containing a hand image by adopting a pre-trained hand detection model, and obtaining position information of a hand detection frame corresponding to the picture; determining a hand region in the picture based on the position information of the hand detection frame; adopting a pre-trained hand key point detection model to detect hand key points contained in the picture of the hand region, and obtaining position information of the hand key points; and performing gesture recognition processing on the position information of the hand key points by adopting a pre-trained gesture recognition model to obtain a gesture recognition result matched with the hand image. According to the method, the consecutive hand detection, hand key point detection and gesture classification processes are adopted, so that the accuracy of hand key point detection is improved, the accuracy of gesture recognition is improved, and the robustness of gesture recognition in a complex environment can be improved.

Description

Technical Field

[0001] The present application relates to the field of gesture recognition technology, and in particular to a gesture recognition method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Gesture recognition, as an important means of human-computer interaction, has been widely used in various fields. Traditional gesture recognition methods primarily rely on simple image processing techniques, such as skin color detection, contour extraction, and motion tracking. These techniques are easily affected by external factors such as lighting conditions and background complexity, which can affect recognition accuracy and stability. In contrast, keypoint-based gesture recognition methods effectively ignore these external interferences by directly identifying key structural points of the hand (such as fingertips, finger joints, and palm positions). Keypoints themselves characterize specific locations in the hand, and the relative relationships between different points can effectively represent hand gestures. With the development of deep learning, keypoint detection methods based on convolutional neural networks (CNNs) have become an effective technology for improving the accuracy and robustness of gesture recognition. High-precision gesture recognition technology is required for environments such as augmented reality (AR), virtual reality (VR), smart homes, and human-computer interaction systems. However, existing gesture recognition methods generally suffer from insufficient recognition accuracy, poor adaptability, and slow response speed, especially in complex backgrounds or with subtle gestures.

[0003] It can be seen that the gesture recognition method in the prior art still needs to be improved. Summary of the Invention

[0004] The embodiments of the present application provide a gesture recognition method and apparatus, which help to improve the accuracy and speed of gesture recognition, and can improve the robustness of gesture recognition.

[0005] In a first aspect, an embodiment of the present application provides a gesture recognition method, comprising:

[0006] Perform hand detection on a picture containing a hand image using a pre-trained hand detection model to obtain a hand detection result corresponding to the picture, wherein the hand detection result includes: position information of a hand detection frame;

[0007] Determining a hand area in the image based on the position information of the hand detection frame;

[0008] Using a pre-trained hand key point detection model to detect hand key points contained in the image of the hand area, and obtaining a hand key point detection result, wherein the hand key point detection result includes: position information of the hand key points;

[0009] A pre-trained gesture recognition model is used to perform gesture recognition processing on the position information of the key points of the hand to obtain a gesture recognition result of the hand image matching.

[0010] In a second aspect, an embodiment of the present application provides a gesture recognition device, comprising:

[0011] A hand detection module is used to perform hand detection on a picture containing a hand image using a pre-trained hand detection model, and obtain a hand detection result corresponding to the picture, wherein the hand detection result includes: position information of a hand detection frame;

[0012] a head region determination module, configured to determine a hand region in the image based on position information of the hand detection frame;

[0013] A hand key point detection module is used to detect the hand key points contained in the image of the hand area using a pre-trained hand key point detection model, and obtain a hand key point detection result, wherein the hand key point detection result includes: position information of the hand key points;

[0014] The gesture recognition module is used to perform gesture recognition processing on the position information of the hand key points using a pre-trained gesture recognition model to obtain a gesture recognition result of the hand image matching.

[0015] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the gesture recognition method described in the embodiment of the present application when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the gesture recognition method disclosed in the embodiment of the present application are implemented.

[0017] The gesture recognition method disclosed in the embodiment of the present application uses a pre-trained hand detection model to perform hand detection on an image containing a hand image, obtaining a hand detection result corresponding to the image, wherein the hand detection result includes: position information of a hand detection frame; based on the position information of the hand detection frame, determining the hand region in the image; using a pre-trained hand key point detection model to detect the hand key points contained in the image in the hand region, obtaining a hand key point detection result, wherein the hand key point detection result includes: position information of the hand key points; using a pre-trained gesture recognition model to perform gesture recognition processing on the position information of the hand key points, obtaining a gesture recognition result matching the hand image. This method improves the accuracy of hand key point detection by adopting a coherent hand detection, hand key point detection, and gesture classification process, thereby improving the precision of gesture recognition and can also improve the robustness of gesture recognition in complex environments.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] Figure 1 This is one of the step flow charts of the gesture recognition method disclosed in the embodiment of this application;

[0021] Figure 2 is a schematic diagram of hand detection results in the gesture recognition method disclosed in an embodiment of the present application;

[0022] Figure 3 Schematic diagram of the effect of enlarging the hand detection frame in the gesture recognition method disclosed in the embodiment of the present application;

[0023] Figure 4 Schematic diagram of the hand key point detection result in the gesture recognition method disclosed in the embodiment of the present application;

[0024] Figure 5 is a schematic diagram of the gesture recognition model structure used in the gesture recognition method disclosed in the embodiment of the present application;

[0025] Figure 6 This is the second step flow chart of the gesture recognition method disclosed in the embodiment of this application;

[0026] Figure 7 2 is a schematic diagram of the effect of searching for the optimal detection frame in the gesture recognition method disclosed in the embodiment of the present application;

[0027] Figure 8 Schematic diagram of the effect of jitter processing on key points in the gesture recognition method disclosed in the embodiment of the present application;

[0028] Figure 9 is a schematic structural diagram of a gesture recognition device disclosed in an embodiment of the present application;

[0029] Figure 10 A block diagram schematically shows an electronic device for executing the method according to the present application; and

[0030] Figure 11 The figure schematically shows a storage unit for storing or carrying a program code for implementing the method according to the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] Reference Figure 1 , a gesture recognition method disclosed in an embodiment of the present application includes: steps 110 to 140.

[0033] Step 110 : Perform hand detection on a picture containing a hand image using a pre-trained hand detection model to obtain a hand detection result corresponding to the picture. The hand detection result includes position information of a hand detection frame.

[0034] Optionally, the hand detection model is trained by the following method: a YOLOv8 network model is trained based on a preset hand classification data set to obtain a hand detection model.

[0035] The specific training method of the hand detection model is described in the following embodiments and will not be repeated here.

[0036] During the gesture recognition phase, after acquiring an image containing a hand image, the image is fed into the hand detection model. The hand detection model then performs hand detection on the image, obtaining a hand detection result. This detection result includes, but is not limited to, the position information of the hand detection frame. This position information indicates the position of the hand detection frame within the image. For example, the position information may be the coordinates of the upper left and lower right corners of the hand detection frame within the image.

[0037] Step 120: Determine the hand area in the image based on the position information of the hand detection frame.

[0038] In some optional embodiments, the position information of the hand detection frame may be used to determine the image region in the image where the hand detection frame is located, and the image region is used as the hand region.

[0039] In other optional embodiments, the determining of the hand area in the picture based on the position information of the hand detection frame includes: enlarging the hand detection frame by a preset proportion based on the position information of the hand detection frame to obtain the position information of the enlarged hand detection frame; and using the picture area corresponding to the position information of the enlarged hand detection frame as the hand area in the picture.

[0040] In actual application, the hand detection frame obtained by the hand detection model may not be able to fully capture the hand area. In order to avoid missing the key points of the hand in subsequent steps, in some embodiments of the present application, the width and height of the hand detection frame obtained by the hand detection model are first magnified by a preset ratio of the original size to expand the effective image area for detecting the key points of the hand. The preset ratio is set based on experience. For example, the preset ratio can be set to 2 / 3. Figure 2 Taking the hand detection frame 210 in the picture shown as an example, the hand area in the picture determined after expansion is as follows Figure 3 As shown in 310.

[0041] Step 130: Use a pre-trained hand key point detection model to detect the hand key points contained in the image of the hand area, and obtain a hand key point detection result, wherein the hand key point detection result includes: position information of the hand key points.

[0042] The training method of the hand key point detection model is described in the relevant embodiments below and will not be repeated here. During the gesture recognition stage, the image of the hand area is input into the hand key point detection model, and the hand key point detection model performs hand key point detection on the input image. The hand key point detection model will output the corresponding hand key point detection results. The hand key point detection results include but are not limited to: the position information of the detected hand key points.

[0043] For example, Figure 3 After the picture shown is input into the hand key point detection model, the hand key point detection model can output the position information of the 21 hand key points detected. Figure 4 As shown, the 21 hand key points include but are not limited to fingertips, knuckles, and wrist key points.

[0044] Step 140 : Perform gesture recognition processing on the position information of the hand key points using a pre-trained gesture recognition model to obtain a gesture recognition result of the hand image matching.

[0045] In some optional embodiments, the position information of the hand key points output by the hand key point detection model is affected by the image acquisition environment and is subject to disturbances. For example, the hand size is larger in images acquired at close range, while the hand size is smaller in images acquired at a distance, resulting in different distances between fingers. On the other hand, constructing the shape of the hand depends on the positional relationship between the hand key points. Therefore, before using a pre-trained gesture recognition model to perform gesture recognition processing on the position information of the hand key points and obtaining the gesture recognition results of the hand image matching, the method further includes: normalizing the position information of the hand key points. For example, the maximum and minimum value normalization method is used to normalize the position coordinates of the hand key points to obtain the normalized position coordinates as the position information of the hand key points.

[0046] Next, the normalized position information of the hand key points is input into a pre-trained gesture recognition model, and the gesture recognition model is used to perform gesture recognition processing on the normalized position information of the hand key points to obtain the gesture recognition result of the hand image matching.

[0047] In some optional embodiments, such as Figure 5As shown, during the training phase, the gesture recognition model includes a feature encoding network 510 and a classification mapping layer 520. The feature encoding network 510 is used to encode the position information of the hand key points input to the gesture recognition model to obtain the feature encoding vectors of the hand key points. The classification mapping layer 520 is used to perform classification mapping on the feature encoding vectors output by the feature encoding network 510 to obtain the gesture category and confidence level of the hand key point matches.

[0048] The training method of the gesture recognition model is described in the following embodiments and will not be repeated here.

[0049] During the application phase, the gesture recognition model can be tailored based on the needs of the gesture recognition application. For example, when gesture recognition is performed using model classification, the feature encoding network 510 and classification mapping layer 520 can be retained. When gesture recognition is performed based on feature matching, only the feature encoding network 510 can be retained.

[0050] Optionally, the gesture recognition model is built based on the KPNet network structure, such as Figure 5 As shown, the feature encoding network 510 of the gesture recognition model further includes: three CBR network modules and two LBR network modules connected in sequence. The CBR network module is composed of a one-dimensional convolutional neural network, a batch normalization layer, and a rectified linear unit connected in sequence; the LBR network module is composed of a linear transformation layer, a batch normalization layer, and a rectified linear unit connected in sequence. The classification mapping layer 520 further includes a LB module and a classification output layer connected in sequence, wherein the LB module is composed of a linear transformation layer and a batch normalization layer connected in sequence; the classification output layer is implemented using a softmax function.

[0051] Optionally, the gesture recognition model first undergoes three CBR module operations to perform feature upscaling on the C-dimensional features of the input N (e.g., N=21) hand key points (e.g., the two-dimensional plane coordinates of the 21 hand key points, the two-dimensional features) and transforms them into encoding vectors of [N,64], [N,128], and [N,512], respectively. Then, the global features of all hand key points are aggregated through a maximum pooling operation. The global feature dimensions are then transformed through two LBR module operations. The output 128-dimensional feature vector is the key point feature encoding of the gesture. Furthermore, the 128-dimensional feature encoding undergoes another LB operation linear transformation and batch normalization processing to transform the dimension to the gesture classification dimension K. Finally, softmax is used to obtain the gesture classification category and confidence.

[0052] Two methods of performing gesture recognition based on hand key points using the gesture recognition model are introduced below.

[0053] In the first method, the gesture recognition model includes: a feature encoding network, and the gesture recognition processing of the position information of the hand key points using a pre-trained gesture recognition model to obtain the gesture recognition result of the hand image matching includes: using the encoding network of the pre-trained gesture recognition model to encode the position information of the hand key points to obtain key point feature coding; performing similarity matching on the key point feature coding with a preset gesture key point feature template to obtain a similarity matching result; and obtaining the gesture recognition result of the hand image matching based on the similarity matching result.

[0054] For example, after the normalized two-dimensional position coordinates of the 21 hand key points are input into the gesture recognition model, the modules of the feature encoding network 510 first perform encoding processing in sequence to obtain the feature encoding of the hand key points; then, the similarity (such as cosine similarity) between the output key point feature encoding and each gesture key point feature template registered for the preset gesture in the preset gesture feature library is calculated, and the gesture key point feature template that matches the key point feature encoding is determined based on the similarity. The preset gesture corresponding to the gesture key point feature template that matches the key point feature encoding is the gesture category matched by the hand key point. Furthermore, the similarity between the gesture key point feature template that matches the feature encoding and the key point feature encoding can be used as the confidence level that the key point feature encoding matches the corresponding gesture category.

[0055] By calculating similarity, the most matching gesture category can be quickly and accurately identified, and a confidence assessment is provided based on this. This mechanism is particularly critical in real-time and high-accuracy scenarios.

[0056] Among them, the gesture key point feature template is obtained by the following method: encoding the position information of the hand key points in the gesture picture of the preset gesture category to obtain the key point feature code; clustering the key point feature code to obtain the key point feature code corresponding to the cluster center as the gesture key point feature template of the preset gesture category.

[0057] Because each hand has a different shape, and there are differences in viewing angles and gesture angles during detection, the features of a single gesture are generally insufficient to fully characterize the characteristics of that type of gesture. In the embodiments of the present application, a clustering method is used to cluster the features of a type of gesture to extract the most representative features, which serve as the gesture key point feature template for that type of gesture. The clustering method can employ, for example, a k-mean clustering method, where the distance used for clustering is the Euclidean distance, and the key point features of the cluster center are used as the most representative features.

[0058] Optionally, you can use pictures of each gesture category in the hand classification dataset disclosed in the prior art, use the hand detection model and hand key point detection model trained in the embodiments of the present application to perform hand detection and hand key point detection in sequence, and then use the gesture classification model to encode the detected hand key points to obtain the key point feature coding corresponding to the gesture in each picture.

[0059] Alternatively, you can collect several images corresponding to custom gestures and then perform hand detection, hand keypoint detection, and hand keypoint encoding according to the aforementioned method to obtain several keypoint feature codes corresponding to the custom gestures. Finally, cluster the keypoint feature codes corresponding to each custom gesture, and use the keypoint feature code at the cluster center as the keypoint feature template for that custom gesture.

[0060] The gesture recognition method disclosed in this application clusters the hand key point feature vectors of specific gesture categories to create hand key point feature templates for each gesture category. When a new gesture needs to be recognized, it is only necessary to capture an image of the new gesture and use the application to perform hand detection, hand key point detection, and hand key point feature extraction and clustering, thereby registering the hand key point feature template for the new gesture. In the subsequent gesture recognition process, the application can then use feature matching to recognize the gesture without performing sample labeling and model training. This simplifies the process of adding new gestures and improves the system's adaptability and generalization to new gestures.

[0061] In the second method, the gesture recognition model includes: a feature encoding network and a classification mapping layer. The gesture recognition model is used to perform gesture recognition processing on the position information of the key points of the hand to obtain the gesture recognition result of the hand image matching, including: using the encoding network of the pre-trained gesture recognition model to encode the position information of the key points of the hand to obtain key point feature encoding; and performing classification mapping on the key point feature encoding through the classification mapping layer to obtain the gesture recognition result of the hand image matching.

[0062] When utilizing the classification capabilities of the gesture recognition model for gesture recognition, the encoding network of the pre-trained gesture recognition model is used, in accordance with the method described above, to encode the positional information of the hand key points. After obtaining key point feature codes, the obtained key point feature codes are input into the classification mapping layer 520, which performs encoding mapping on the key point feature codes to obtain the confidence level that the hand key points match the preset gesture category. Furthermore, the preset gesture category to which the hand key points match can be determined based on the maximum confidence level for matching the preset gesture category. For example, when the maximum confidence level is greater than a preset confidence threshold, the preset gesture category corresponding to the maximum confidence level is the gesture category to which the hand key points match.

[0063] When implementing it specifically, Figure 6 As shown, the method further includes: step 100 before performing hand detection on a picture containing a hand image using a pre-trained hand detection model and obtaining a hand detection result corresponding to the picture.

[0064] Step 100: train the hand detection model, the hand key point detection model, and the gesture recognition model in sequence.

[0065] The specific training methods of each model are described below.

[0066] (1) Hand detection model

[0067] In some embodiments of the present application, an existing hand classification dataset may be used to train a YOLOv8 hand detection model.

[0068] Each sample image in the hand classification dataset contains a label: gesture category and the position information of the hand detection box. The category is represented by cls (cls is an integer), and the position information of the hand detection box is represented by the upper left corner coordinates and the lower right corner coordinates. The coordinate format is (x_min, y_min, x_max, y_max), where x_min, y_min represent the upper left corner coordinates of the hand detection box, and x_max, y_max represent the lower right corner coordinates of the hand detection box.

[0069] In specific implementation, the coordinates of the upper left corner and lower right corner of the hand detection box are converted to the normalized coordinate format used by YOLO, that is, the normalized center point coordinates and width and height are expressed in the coordinate format of (x_cn, y_cn, w_n, h_n), so as to regress the prediction results to within 0-1.

[0070] For example, the upper left corner coordinates and lower right corner coordinates of the hand detection box can be normalized using the following formula: Among them, x_cn, y_cn represent the normalized center coordinates of the hand detection box, w_n, h_n represent the normalized width and height of the hand detection box respectively, W represents the width of the sample image, and H represents the height of the sample image.

[0071] Afterwards, the gesture category label and the normalized position information of the hand detection frame of the sample image are used as the new sample label of the sample image to generate a new hand classification dataset. The YOLOv8 model is trained based on the generated new hand classification dataset to obtain the hand detection model.

[0072] In other optional embodiments, a YOLOv8 variant may be used, or other deep learning models such as Faster R-CNN, SSD, or Mask R-CNN may be used to train a hand detection model, which are not listed one by one in the embodiments of this application.

[0073] (2) Hand key point detection model

[0074] In some optional embodiments, MediaPipe can be used to annotate hand key points on sample images in a hand classification dataset for training a hand detection model, thereby generating a dataset for training a hand key point detection model.

[0075] The sample images in the preset hand classification dataset are provided with a hand detection frame label and a gesture category label, and the hand key point detection model is trained by the following method: determining an initial detection frame based on the hand detection frame label; performing multiple rounds of magnification on the image area corresponding to the initial detection frame, and predicting hand key points for the sample image content in the magnified area obtained after each round of magnification, and heuristically searching for hand key points in the sample image; enlarging the initial detection frame to obtain a screenshot area in the sample image; capturing the screenshot area in the sample image as a reconstructed sample image; annotating the reconstructed sample image with the gesture category label and the searched hand key points to construct a hand key point detection dataset; and training the hand key point detection model based on the hand key point detection dataset. The hand detection frame label is used to describe the position of the hand detection frame in the sample image; and the gesture category label is used to indicate the gesture category matched by the hand image in the corresponding sample image.

[0076] When detecting hand key points, if the image lacks background information or introduces too much background information or invalid information, it will lead to hand key point prediction failure or missed detection. In the embodiment of the present application, to solve the above problem, the key point detection area is first dynamically adjusted according to the hand detection box in the preset hand classification dataset to search for the optimal key point detection area. The image content within the optimal key point detection area contains sufficient header key point information and background information, while eliminating redundant background information.

[0077] Optionally, based on the hand detection frame label, an initial detection frame is determined; by performing multiple rounds of magnification on the image area corresponding to the initial detection frame, and predicting hand key points of the sample image content in the magnified area obtained after each round of magnification, a heuristic search is performed for hand key points in the sample image, including the following sub-steps S1 to S7.

[0078] Sub-step S1, initializing search parameters, the search parameters including: expansion coefficient, current iteration round, expansion coefficient iteration increment, target intersection-over-union ratio.

[0079] The expansion coefficient a_start is initialized to 1.5, the current iteration number n is initialized to 0, the expansion coefficient iteration increment p is initialized to 0.5, the best intersection-over-union ratio iou_best is initialized to 0.0, and the initial hand detection box is represented as bbox. The initial hand detection box is the hand detection box in the sample label.

[0080] Sub-step S2, obtaining the current round expansion coefficient.

[0081] Optionally, the current round expansion coefficient may be calculated using the formula a_now=a_start+n*p, where a_now represents the current round expansion coefficient.

[0082] Sub-step S3: enlarge the initial hand detection frame based on the current round enlargement coefficient to obtain an enlarged detection frame.

[0083] For example, the width and height of the initial hand detection frame are expanded by a_now times respectively.

[0084] Sub-step S4: performing hand key point detection based on the image content in the expanded detection frame to obtain a key point frame bbox_kp.

[0085] Optionally, use Meidapipe to perform hand key point detection on the image area content within the expanded detection frame in the sample image, predict the coordinates of the hand key points included in the area, and then use the minimum horizontal coordinate and the minimum vertical coordinate, as well as the maximum horizontal coordinate and the maximum vertical coordinate of the predicted hand key point coordinates as the diagonal vertices of the rectangle to determine the rectangular box as the key point box bbox_kp.

[0086] Sub-step S5, calculating the current intersection-over-union ratio of the initial hand detection box bbox and the key point box bbox_kp.

[0087] Optionally, the current intersection-of-union ratio iou_now of the initial hand detection box bbox and the key point box bbox_kp is the ratio of the area of ​​the overlapping region of the two to the sum of the areas of the two minus the overlapping area.

[0088] In sub-step S6, when the current IoU is less than or equal to the target IoU, jump to execute sub-step S7; when the current IoU is greater than the target IoU and the preset iteration round has not been reached, update the target IoU to the current IoU, jump to sub-step S2, and execute the next round of iterative adjustment; when the current IoU is greater than the target IoU and the preset iteration round has not been reached, it is considered that the search for the hand key point has failed.

[0089] Sub-step S7: When the updated target IoU is greater than a preset IoU threshold, the key point frame determined in the current round of iteration is used as the preferred detection frame.

[0090] The preset IoU threshold is set based on experience, for example, the preset IoU threshold can be set to 0.65. If the target IoU is less than or equal to the preset IoU threshold, the hand key points predicted in the current iteration are considered invalid and the iteration is stopped.

[0091] The effect of optimizing the detection frame is as follows Figure 7 As shown, the entire image area is the size of the expanded detection frame. MediaPipe will be used to predict the hand key points in this area. The solid-line frame is the hand detection frame in the sample label, and the dotted-line frame is the key point frame bbox_kp obtained based on the predicted hand key point coordinates. The intersection-over-union ratio (iou_now) reflects the degree of overlap between the predicted key point bounding box and the hand detection frame in the sample label. If the two completely overlap, the intersection-over-union ratio (iou_now) is equal to 1. Because the hand area is composed of fingers and palms, there must be key points at each joint. If the coverage area is small, it means that there are missed detections or false detections. Multiple iterations use a heuristic search method to find the optimal solution for the detection frame with the fewest possible detection times and return as many valid detection results as possible.

[0092] In an embodiment of the present application, the accuracy and reliability of hand key point prediction are significantly improved by dynamically adjusting the width and height of the detection area and using heuristic search methods (such as expansion coefficients and iterative optimization) to search for the optimal hand key points for constructing training data.

[0093] Considering that actual hand detection may be incomplete and requires certain background information, when constructing the hand key point detection dataset, the original detection bounding box (bbox) is magnified by 2 / 3 to obtain a screenshot area in the sample image. The content of the sample image is captured according to the screenshot area and used as the reconstructed sample image. The reconstructed sample image is then used as the sample data for the training samples in the hand key point detection dataset. The location information of the searched optimal hand key point is used as the hand key point label, and the gesture category label of the original sample image is retained to generate the training samples in the hand key point detection dataset.

[0094] Optionally, the training the hand key point detection model based on the hand key point detection dataset includes: training a YOLOv8-pose model based on the hand key point detection dataset to obtain a hand key point detection model.

[0095] The YOLOv8-pose model is trained based on the hand key point detection dataset. The specific implementation method of obtaining the hand key point detection model can be found in the prior art and will not be repeated in the embodiments of this application.

[0096] In the embodiments of the present application, the hand key point detection model can also adopt other network model structures for key point detection in the prior art, which will not be listed one by one in the embodiments of the present application.

[0097] In an embodiment of the present application, MediaPipe is used to automate the process of hand key point annotation, and the automatically generated data is used for dual-model training of the hand detection model and the key point detection model. This reduces the need for manual annotation and improves the efficiency of dataset preparation.

[0098] (3) Gesture Recognition Model

[0099] In some embodiments of the present application, the aforementioned hand key point detection dataset can be used to generate a gesture classification key point dataset. A network model specifically for identifying and classifying hand motion key points can be trained, such as Figure 5 The gesture recognition model shown in Figure 1. After removing the last classification layer of the network, the network can output multi-dimensional feature vectors that capture the detailed features of the gesture key points, providing a basis for subsequent feature matching and clustering analysis.

[0100] In some optional embodiments, the gesture recognition model is trained by the following method: using the image content in the preferred detection frame of the reconstructed sample image in the hand key point detection dataset as the gesture sample image and performing random jitter processing on the marked hand key points within a preset range to obtain a gesture classification key point dataset; based on the gesture classification key point dataset, the gesture recognition model is trained. The gesture recognition model only needs to obtain the hand key points, so only the area of ​​the key point box bbox_kp is used. In order to increase robustness and generalization ability, the key point coordinates can be normalized using maximum and minimum values, and random jitter within the range of ±5% can be added to each key point during the training phase. The effect of random jitter processing within a preset range on the hand key points is as follows. Figure 8 shown.

[0101] Optionally, in the process of training the gesture recognition model based on the gesture classification key point dataset, the model loss function of the gesture recognition model is configured to include: encoding loss and classification loss, wherein the encoding loss is: the maximum difference between the positive sample prediction error and the negative sample prediction error, wherein the positive sample prediction error is: the square of the Euclidean distance between the anchor point and the positive sample, and the negative sample prediction error is: the square of the Euclidean distance between the anchor point and the negative sample.

[0102] Taking the structure of the gesture recognition model described above as an example, when training the gesture recognition model, a triplet loss is introduced to the first layer of the classification (dimension [1,128]) to increase the differentiation between different gestures. Given three 128-dimensional feature vectors fa, fp, and fn corresponding to the anchor point, positive sample, and negative sample features respectively, the triplet loss L is defined as:

[0103] L=max(0,‖f a -f p ‖ 2 -‖f a -f n ‖ 2 +margin);

[0104] Among them, ‖f a -f p ‖ 2 represents the square of the Euclidean distance between the anchor point and the positive sample, ‖f a -f n ‖ 2 Represents the square of the Euclidean distance between the anchor point and the negative sample. Margin is a hyperparameter used to define the minimum distance between positive and negative samples, for example, the value is 1.

[0105] In the embodiments of the present application, an anchor point refers to a feature code of a gesture image, a positive sample refers to a feature code of an image with the same gesture as that in the anchor point image, and a negative sample refers to a feature code of an image with a different gesture from that in the anchor point image.

[0106] With the goal of minimizing the model loss L, the parameters of the feature encoding network 510 of the gesture recognition model are optimized.

[0107] The gesture recognition method disclosed in the embodiments of the present application introduces triplet loss in the model training stage to enhance the distinction between different gestures, thereby improving the classification effect of the model and the discriminative power of feature representation.

[0108] On the other hand, the loss function of the gesture recognition model also includes classification loss. The calculation method of gesture classification loss refers to the model loss function of the classification network in the prior art. The embodiment of this application does not limit the calculation method of classification loss.

[0109] After iteratively training the gesture recognition model by optimizing the model loss, the trained gesture recognition model is deployed in an application scenario. In actual applications, the gesture recognition model can be used solely with the feature encoding network 510 to extract features from input key points and obtain key point feature codes. Alternatively, the feature encoding network 510 and the classification mapping layer 520 can be combined to first extract features from the input key points, then perform feature mapping on the extracted key point feature codes, ultimately obtaining a gesture classification result.

[0110] This application utilizes a gesture recognition model to not only encode but also classify hand key points. After removing the final classification layer, the model outputs a multidimensional feature vector, which provides the basis for subsequent hand key point feature matching and cluster analysis. This approach offers significant advantages in improving the accuracy and flexibility of gesture recognition.

[0111] In summary, the gesture recognition method disclosed in the embodiments of the present application uses a pre-trained hand detection model to perform hand detection on a picture containing a hand image, and obtains a hand detection result corresponding to the picture, wherein the hand detection result includes: position information of the hand detection frame; based on the position information of the hand detection frame, the hand area in the picture is determined; a pre-trained hand key point detection model is used to detect the hand key points contained in the picture in the hand area, and obtains a hand key point detection result, wherein the hand key point detection result includes: position information of the hand key points; a pre-trained gesture recognition model is used to perform gesture recognition processing on the position information of the hand key points, and obtains a gesture recognition result matching the hand image. This method improves the accuracy of hand key point detection by adopting a coherent hand detection, hand key point detection, and gesture classification process, thereby improving the accuracy of gesture recognition, and can also improve the robustness of gesture recognition in complex environments.

[0112] When performing gesture recognition, by optimizing the network structure of the gesture recognition model, feature encoding is performed directly after key point detection, and gesture recognition is performed using feature matching, which significantly reduces processing time and makes the entire gesture recognition process faster.

[0113] Reference Figure 9 , the embodiment of the present application further discloses a gesture recognition device, the device comprising:

[0114] The hand detection module 910 is configured to perform hand detection on a picture containing a hand image using a pre-trained hand detection model, and obtain a hand detection result corresponding to the picture, wherein the hand detection result includes: position information of a hand detection frame;

[0115] a head region determination module 920, configured to determine a hand region in the image based on the position information of the hand detection frame;

[0116] The hand key point detection module 930 is configured to detect the hand key points contained in the image of the hand region using a pre-trained hand key point detection model, and obtain a hand key point detection result, wherein the hand key point detection result includes: position information of the hand key points;

[0117] The gesture recognition module 940 is configured to perform gesture recognition processing on the position information of the hand key points using a pre-trained gesture recognition model to obtain a gesture recognition result of the hand image matching.

[0118] Optionally, the gesture recognition model includes: a feature encoding network, wherein the gesture recognition processing is performed on the position information of the key points of the hand using a pre-trained gesture recognition model to obtain the gesture recognition result of the hand image matching, including:

[0119] Using the encoding network of the pre-trained gesture recognition model to encode the position information of the key points of the hand to obtain key point feature codes;

[0120] Performing similarity matching between the key point feature code and a preset gesture key point feature template to obtain a similarity matching result;

[0121] Based on the similarity matching result, a gesture recognition result of the hand image matching is obtained.

[0122] Optionally, the gesture recognition model includes: a feature encoding network and a classification mapping layer. The pre-trained gesture recognition model is used to perform gesture recognition processing on the position information of the hand key points to obtain the gesture recognition result of the hand image matching, including:

[0123] Using the encoding network of the pre-trained gesture recognition model to encode the position information of the key points of the hand to obtain key point feature codes;

[0124] The key point feature codes are classified and mapped through the classification mapping layer to obtain the gesture recognition result of the hand image matching.

[0125] Optionally, determining the hand area in the image based on the position information of the hand detection frame includes:

[0126] performing an enlargement process on the hand detection frame by a preset ratio based on the position information of the hand detection frame to obtain position information of the hand detection frame after the enlargement process;

[0127] The image area corresponding to the position information of the expanded hand detection frame is used as the hand area in the image.

[0128] Optionally, the sample images in the preset hand classification dataset are provided with hand detection frame labels and gesture category labels, and the hand key point detection model is trained by the following method:

[0129] Determining an initial detection frame based on the hand detection frame label;

[0130] Performing multiple rounds of magnification on the image area corresponding to the initial detection frame, and predicting hand key points on the sample image content in the magnified area obtained after each round of magnification, and heuristically searching for hand key points in the sample image;

[0131] Expanding the initial detection frame to obtain a screenshot area in the sample image;

[0132] intercepting the screenshot area in the sample image as a reconstructed sample image;

[0133] Annotate the reconstructed sample images with the gesture category labels and the hand key points obtained by the search, and construct a hand key point detection dataset;

[0134] The hand key point detection model is trained based on the hand key point detection dataset.

[0135] Optionally, the gesture recognition model is trained by the following method:

[0136] Using the image content in the preferred detection frame of the reconstructed sample image in the hand key point detection dataset as a gesture sample image and performing random jitter processing on the marked hand key points within a preset range to obtain a gesture classification key point dataset;

[0137] The gesture recognition model is trained based on the gesture classification key point dataset.

[0138] The gesture recognition device disclosed in the embodiments of the present application is used to implement the gesture recognition method described in the embodiments of the present application. The specific implementation of each module of the device will not be repeated here. Please refer to the specific implementation of the corresponding steps in the method embodiment.

[0139] The gesture recognition device disclosed in the embodiment of the present application performs hand detection on an image containing a hand image using a pre-trained hand detection model to obtain a hand detection result corresponding to the image, wherein the hand detection result includes: position information of a hand detection frame; based on the position information of the hand detection frame, the hand region in the image is determined; a pre-trained hand key point detection model is used to detect the hand key points contained in the image in the hand region to obtain a hand key point detection result, wherein the hand key point detection result includes: position information of the hand key points; a pre-trained gesture recognition model is used to perform gesture recognition processing on the position information of the hand key points to obtain a gesture recognition result matching the hand image. By adopting a coherent hand detection, hand key point detection, and gesture classification process, the device improves the accuracy of hand key point detection, thereby improving the precision of gesture recognition, and can also improve the robustness of gesture recognition in complex environments.

[0140] When performing gesture recognition, by optimizing the network structure of the gesture recognition model, feature encoding is performed directly after key point detection, and gesture recognition is performed using feature matching, which significantly reduces processing time and makes the entire gesture recognition process faster.

[0141] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between the various embodiments can be referred to in conjunction with each other. For the device embodiments, since they are generally similar to the method embodiments, their description is relatively simple, and for relevant parts, reference can be made to the description of the method embodiments.

[0142] The above is a detailed introduction to a gesture recognition method and device provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0144] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to the embodiment of the present application. The application can also be implemented as a device or apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0145] For example, Figure 10An electronic device that can implement the method according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device conventionally includes a processor 1010 and a memory 1020, and program code 1030 stored on the memory 1020 and executable on the processor 1010. When the processor 1010 executes the program code 1030, the method described in the above embodiments is implemented. The memory 1020 can be a computer program product or a computer-readable medium. The memory 1020 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. The memory 1020 has a storage space 10201 for program code 1030 of a computer program for executing any of the method steps described above. For example, the storage space 10201 for program code 1030 can include individual computer programs for implementing various steps in the above method. The program code 1030 is computer-readable code. These computer programs can be read from or written into one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. The computer program includes computer readable code that, when run on an electronic device, causes the electronic device to perform the method according to the above embodiments.

[0146] The embodiment of the present application further discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the gesture recognition method described in the embodiment of the present application are implemented.

[0147] Such a computer program product may be a computer-readable storage medium having a computer program product. Figure 10 The memory 1020 in the electronic device shown is similarly arranged as a storage segment, storage space, etc. The program code can be compressed and stored in the computer readable storage medium in an appropriate form. The computer readable storage medium is generally as shown in FIG. Figure 11 The portable or fixed storage unit generally includes computer-readable code 1030', which is a code read by a processor and implements the steps of the above-described method when executed by the processor.

[0148] References herein to "one embodiment," "an embodiment," or "one or more embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Furthermore, please note that instances of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0149] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0150] In the claims, any reference signs placed between brackets shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A gesture recognition method, characterized in that: The method comprises: Perform hand detection on a picture containing a hand image using a pre-trained hand detection model to obtain a hand detection result corresponding to the picture, wherein the hand detection result includes: position information of a hand detection frame; Determining a hand area in the image based on the position information of the hand detection frame; Using a pre-trained hand key point detection model to detect hand key points contained in the image of the hand area, and obtaining a hand key point detection result, wherein the hand key point detection result includes: position information of the hand key points; A pre-trained gesture recognition model is used to perform gesture recognition processing on the position information of the key points of the hand to obtain a gesture recognition result of the hand image matching.

2. The method according to claim 1, characterized in that The gesture recognition model includes: a feature encoding network, wherein the gesture recognition processing is performed on the position information of the key points of the hand using the pre-trained gesture recognition model to obtain the gesture recognition result of the hand image matching, including: Using the encoding network of the pre-trained gesture recognition model to encode the position information of the key points of the hand to obtain key point feature codes; Performing similarity matching between the key point feature code and a preset gesture key point feature template to obtain a similarity matching result; Based on the similarity matching result, a gesture recognition result of the hand image matching is obtained.

3. The method according to claim 1, characterized in that The gesture recognition model includes: a feature encoding network and a classification mapping layer. The gesture recognition model is pre-trained to perform gesture recognition processing on the position information of the key points of the hand to obtain the gesture recognition result of the hand image matching, including: Using the encoding network of the pre-trained gesture recognition model to encode the position information of the key points of the hand to obtain key point feature codes; The key point feature codes are classified and mapped through the classification mapping layer to obtain the gesture recognition result of the hand image matching.

4. The method according to claim 1, wherein The determining the hand area in the image based on the position information of the hand detection frame includes: performing an enlargement process on the hand detection frame by a preset ratio based on the position information of the hand detection frame to obtain position information of the hand detection frame after the enlargement process; The image area corresponding to the position information of the expanded hand detection frame is used as the hand area in the image.

5. The method according to claim 1, wherein The sample images in the preset hand classification dataset are provided with hand detection frame labels and gesture category labels, and the hand key point detection model is trained by the following method: Determining an initial detection frame based on the hand detection frame label; Performing multiple rounds of magnification on the image area corresponding to the initial detection frame, and predicting hand key points on the sample image content in the magnified area obtained after each round of magnification, and heuristically searching for hand key points in the sample image; Expanding the initial detection frame to obtain a screenshot area in the sample image; intercepting the screenshot area in the sample image as a reconstructed sample image; Annotate the reconstructed sample images with the gesture category labels and the hand key points obtained by the search, and construct a hand key point detection dataset; The hand key point detection model is trained based on the hand key point detection dataset.

6. The method according to claim 5, characterized in that The gesture recognition model is trained by the following method: Using the image content in the preferred detection frame of the reconstructed sample image in the hand key point detection dataset as a gesture sample image and performing random jitter processing on the marked hand key points within a preset range to obtain a gesture classification key point dataset; The gesture recognition model is trained based on the gesture classification key point dataset.

7. A gesture recognition device, characterized in that: The device comprises: A hand detection module is used to perform hand detection on a picture containing a hand image using a pre-trained hand detection model, and obtain a hand detection result corresponding to the picture, wherein the hand detection result includes: position information of a hand detection frame; a head region determination module, configured to determine a hand region in the image based on position information of the hand detection frame; A hand key point detection module is used to detect the hand key points contained in the image of the hand area using a pre-trained hand key point detection model, and obtain a hand key point detection result, wherein the hand key point detection result includes: position information of the hand key points; The gesture recognition module is used to perform gesture recognition processing on the position information of the hand key points using a pre-trained gesture recognition model to obtain a gesture recognition result of the hand image matching.

8. The device according to claim 7, characterized in that The sample images in the preset hand classification dataset are provided with hand detection frame labels and gesture category labels, and the hand key point detection model is trained by the following method: Determining an initial detection frame based on the hand detection frame label; Performing multiple rounds of magnification on the image area corresponding to the initial detection frame, and predicting hand key points on the sample image content in the magnified area obtained after each round of magnification, and heuristically searching for hand key points in the sample image; Expanding the initial detection frame to obtain a screenshot area in the sample image; intercepting the screenshot area in the sample image as a reconstructed sample image; Annotate the reconstructed sample images with the gesture category labels and the hand key points obtained by the search, and construct a hand key point detection dataset; The hand key point detection model is trained based on the hand key point detection dataset.

9. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, wherein: When the processor executes the program code, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having program code stored thereon, characterized in that: When the program code is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images