Gesture recognition method, apparatus and system, and storage medium

Abstract

Provided are a gesture recognition method, apparatus and system, and a storage medium. The gesture recognition method comprises: matching a gesture feature to be recognized with gesture features in a gesture feature library to determine a first matching set (101); determining a target gesture feature in the first matching set, and matching the target gesture feature with gesture features in the gesture feature library to determine a second matching set (102); determining a third matching set according to the first matching set and the second matching set (103); and matching the gesture feature to be recognized with gesture features in the third matching set to determine a gesture recognition result for the gesture feature to be recognized (104). By means of a plurality of matching processes, missing recognition of a gesture feature to be recognized is avoided, thereby improving the gesture recognition precision.

Description

Gesture recognition method, device, system and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on March 8, 2023, with application number 2023102439597 and invention name “Gesture Recognition Method, Device, System and Storage Medium”, the contents of which should be understood as incorporated into this application by reference. Technical Field

[0002] This article relates to, but is not limited to, the field of image recognition technology, and in particular to a gesture recognition method, device, system, and storage medium. Background Art

[0003] With the development of artificial intelligence, gesture recognition, as an emerging human-computer interaction technology, has been applied to an increasing number of fields. Gesture recognition technology frees users from the constraints of physical input devices. For example, users can use gestures to take photos and swipe on their phones; they can control smart home appliances like lights, curtains, and air conditioners; and they can control car windows and adjust the volume. With its unique advantages of contactless operation and "what you do is what you get," gesture recognition technology brings tremendous convenience to users and is also changing the traditional model of human-computer interaction.

[0004] Gesture recognition technology can capture images of a user's hand, identify gestures within them, and determine interaction commands, thereby executing the corresponding actions. However, low accuracy in gesture recognition can lead to delayed or erroneous responses.

[0005] Summary of the Invention

[0006] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0007] The embodiments of the present application provide a gesture recognition method, device, system, and storage medium, which can improve the accuracy of gesture recognition.

[0008] An embodiment of the present application provides a gesture recognition method, the method comprising:

[0009] Matching the gesture features to be recognized with the gesture features in the gesture feature library to determine a first matching set;

[0010] Determining a target gesture feature in the first matching set, matching the target gesture feature with gesture features in the gesture feature library, and determining a second matching set;

[0011] determining a third matching set based on the first matching set and the second matching set;

[0012] The gesture feature to be recognized is matched with gesture features in the third matching set to determine a gesture recognition result of the gesture feature to be recognized.

[0013] In some embodiments, the method further comprises:

[0014] Determining the number of intersection features according to the intersection of the first matching set and the second matching set;

[0015] When the number of the intersection features meets a preset set number threshold, the third matching set is determined according to the first matching set and the second matching set.

[0016] In some embodiments, determining a third matching set based on the first matching set and the second matching set includes:

[0017] The union of the first matching set and the second matching set is used as the third matching set.

[0018] In some embodiments, matching the gesture features to be recognized with the gesture features in the gesture feature library to determine a first matching set includes:

[0019] Determining the first matching set based on the similarity between the gesture feature to be recognized and the gesture features in the gesture feature library and a preset first matching parameter;

[0020] and / or,

[0021] Matching the target gesture feature with the gesture features in the gesture feature library to determine a second matching set includes:

[0022] The second matching set is determined based on the similarity between the target gesture feature and the gesture features in the gesture feature library and a preset second matching parameter.

[0023] In some embodiments, the first matching parameter includes a first quantity parameter, the second matching parameter includes a second quantity parameter, and the second quantity parameter is smaller than the first quantity parameter.

[0024] In some embodiments, the similarity between the gesture feature to be recognized and the gesture features in the gesture feature library is calculated using a k-nearest neighbor algorithm.

[0025] In some embodiments, the similarity between the target gesture feature and the gesture features in the gesture feature library is calculated using a k-nearest neighbor algorithm, and the k value used to calculate the second matching set is smaller than the k value used to calculate the first matching set.

[0026] In some embodiments, matching the gesture features to be recognized with the gesture features in the gesture feature library to determine a first matching set includes:

[0027] Performing an initial match between the gesture feature to be identified and the gesture features in the gesture feature library, and if the initial match is successful, adding the gesture feature to be identified to the gesture feature library to obtain an expanded gesture feature library;

[0028] The gesture features to be recognized are matched again with the gesture features in the expanded gesture feature library to obtain the first matching set.

[0029] In some embodiments, the method further comprises:

[0030] The gesture feature library includes multiple types of gesture features. When the number of gesture features of a target type in the gesture feature library is greater than a preset storage number, the multiple gesture features of the target type are screened.

[0031] In some embodiments, screening the plurality of gesture features of the target type includes:

[0032] The multiple gesture features of the target type are screened according to at least one of a preset similarity and an acquisition time of each gesture feature in the gesture feature library.

[0033] In some embodiments, the method for acquiring the gesture feature to be recognized includes:

[0034] Obtain the shape features and skeleton point features of the gesture in the image to be detected;

[0035] The gesture feature to be recognized is determined based on the appearance feature and the skeleton point feature.

[0036] In some embodiments, the skeleton point feature is a 2.5D feature.

[0037] In some embodiments, the method further comprises:

[0038] Obtaining the registered gesture features in the registered image;

[0039] The registered gesture feature is registered in the gesture feature library according to the registered gesture feature and the gesture function corresponding to the registered gesture feature.

[0040] An embodiment of the present application provides a gesture recognition device, the device comprising a first determination module, a second determination module, a third determination module, and a fourth determination module:

[0041] The first determination module is configured to match the gesture feature to be recognized with the gesture features in the gesture feature library to determine a first matching set;

[0042] The second determination module is configured to determine a target gesture feature in the first matching set, match the target gesture feature with gesture features in the gesture feature library, and determine a second matching set;

[0043] The third determination module is configured to determine a third matching set based on the first matching set and the second matching set;

[0044] The fourth determination module is configured to match the gesture feature to be recognized with gesture features in the third matching set to determine a gesture recognition result of the gesture feature to be recognized.

[0045] The embodiment of the present application provides a gesture recognition system, including an image acquisition device and a gesture recognition device:

[0046] The image acquisition device is configured to acquire features of the gesture to be recognized;

[0047] The gesture recognition device is configured to match the gesture feature to be recognized with the gesture features in a gesture feature library to determine a first matching set; determine a target gesture feature in the first matching set, match the target gesture feature with the gesture features in the gesture feature library to determine a second matching set; determine a third matching set based on the first matching set and the second matching set; match the gesture feature to be recognized with the gesture features in the third matching set to determine a gesture recognition result of the gesture feature to be recognized.

[0048] An embodiment of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement any of the above-mentioned gesture recognition methods.

[0049] An embodiment of the present application provides a gesture recognition device, including a memory and a processor. The memory stores a program, and when the program is read and executed by the processor, it implements any of the above-mentioned gesture recognition methods.

[0050] Compared to related technologies, the embodiments of this application include: matching the gesture features to be recognized with gesture features in a gesture feature library to determine a first matching set; determining target gesture features in the first matching set, matching the target gesture features with gesture features in the gesture feature library to determine a second matching set; determining a third matching set based on the first matching set and the second matching set; and matching the gesture features to be recognized with gesture features in the third matching set to determine a gesture recognition result for the gesture features to be recognized. Through multiple matching processes, missed recognition of the gesture features to be recognized is avoided, improving the accuracy of gesture recognition.

[0051] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.

[0052] Summary of the Figures

[0053] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0054] FIG1 is a flowchart of a gesture recognition method according to an embodiment of the present application;

[0055] FIG2 is a flowchart of a method for determining a third matching set according to an embodiment of the present application;

[0056] FIG3 is a flow chart of a method for acquiring features of a gesture to be recognized according to an embodiment of the present application;

[0057] FIG4 is a flowchart of a gesture registration method according to an embodiment of the present application;

[0058] FIG5 is a flow chart of a network training method according to an embodiment of the present application;

[0059] FIG6 is a schematic diagram of expanding a gesture feature library according to an embodiment of the present application;

[0060] FIG7 is a schematic diagram of a matching process according to an embodiment of the present application;

[0061] FIG8 is a structural block diagram of a gesture recognition device according to an embodiment of the present application.

[0062] Details

[0063] This application describes multiple embodiments, but this description is exemplary rather than restrictive, and it will be apparent to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described herein. Although many possible feature combinations are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0064] This application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features, and elements disclosed in this application may also be combined with any conventional features or elements to form a unique inventive solution defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other inventive solutions to form another unique inventive solution defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in this application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the appended claims and their equivalents, the embodiments are not subject to other limitations. In addition, various modifications and changes may be made within the scope of protection of the appended claims.

[0065] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be understood by those skilled in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to performing their steps in the order written, and those skilled in the art can readily understand that these orders can be changed and still remain within the spirit and scope of the embodiments of the present application.

[0066] The present invention provides a gesture recognition method, as shown in FIG1 , which includes the following steps:

[0067] Step S101, matching the gesture features to be recognized with the gesture features in the gesture feature library to determine a first matching set;

[0068] In this embodiment, the gesture features to be recognized can be directly obtained or detected from the acquired image. The gesture features to be recognized can be of various types, for example, any gesture features that appear in various scenarios such as controlling mobile phones, vehicles, smart homes, augmented reality (AR), and virtual reality (VR).

[0069] The gesture feature library may be a set of preset multiple types of gesture features, and the number of gesture features of each type may be one or more. The process of matching the gesture feature to be identified with the gesture features in the gesture feature library may be achieved by at least one of feature point detection and similarity calculation;

[0070] In order to avoid missing recognition of gesture features to be recognized during the recognition process, this embodiment selects multiple gesture features from the gesture feature library to form a set as the recognition result of the gesture features to be recognized; wherein, the selection basis can be preset parameters such as similarity and acquisition time;

[0071] Step S102, determining a target gesture feature in the first matching set, matching the target gesture feature with gesture features in the gesture feature library, and determining a second matching set;

[0072] In order to avoid missed recognition, in this embodiment, one or more gesture features in the first matching set are sequentially used as target gesture features and matched again with the gesture features in the gesture feature library; if there are multiple target gesture features, multiple second matching sets corresponding to the multiple target gesture features will be obtained.

[0073] In this embodiment, there is no limit on the number of target gesture features. One or more gesture features may be randomly selected from the first matching set, or one or more gesture features may be selected according to a preset rule. For example, all gesture features in the first matching set may be sequentially used as target gesture features.

[0074] Step S103: Determine a third matching set according to the first matching set and the second matching set.

[0075] After obtaining the first matching set and the second matching set, a third matching set may be determined based on the gesture features in the first matching set and the second matching set. For example, the third matching set may be determined based on the union or intersection of the first matching set and the second matching set, or based on the similarity between the gesture features in the first matching set and the second matching set and the gesture features to be recognized, some gesture features may be selected from the two sets to form the third matching set.

[0076] When there is only one target gesture feature, the third matching set can be determined directly based on the first matching set and the second matching set;

[0077] In the case of multiple target gesture features, there are multiple second matching sets corresponding to the multiple target gesture features. In this case, the third matching set can be directly determined based on the first matching set and all the second matching sets, or each second matching set can be compared with the first matching set one-to-one to obtain multiple comparison results, and then the third matching set can be determined based on the multiple comparison results.

[0078] Step S104, matching the gesture feature to be recognized with the gesture features in the third matching set to determine a gesture recognition result of the gesture feature to be recognized;

[0079] In the embodiment of the present application, the third matching set is used as the set that the gesture features to be recognized ultimately need to be matched to perform feature matching, thereby obtaining a gesture recognition result for the gesture features to be recognized. For example, the gesture recognition result in this embodiment is the control function corresponding to the gesture features to be recognized.

[0080] This embodiment does not impose any requirements on the implementation method of each matching stage. In terms of matching accuracy, because as many similar gestures as possible are covered in the process of obtaining the first matching set and the second matching set, the matching accuracy can be set lower, and the matching algorithm used can be a basic cosine distance, Euclidean distance, Mahalanobis distance, etc.; in the process of determining the gesture recognition result of the gesture feature to be recognized, in order to improve the recognition accuracy, the matching accuracy can be set higher, and the matching algorithm used can be a more accurate algorithm, such as the Jaccard distance.

[0081] Through the above steps S101 to S104, this embodiment improves the recall rate of the gesture recognition process based on the multiple matching processes between the gesture features to be recognized and the gesture features in the gesture feature library, avoids the missed recognition of the gesture features to be recognized, and improves the accuracy of gesture recognition.

[0082] In some embodiments, a flowchart of a method for determining a third matching set is shown in FIG2 . The method includes the following steps:

[0083] Step S201: determining the number of intersection features according to the intersection of the first matching set and the second matching set;

[0084] The more the number of intersection features between the first matching set and the second matching set, the more similar the gesture features in the two sets are, which is conducive to improving the accuracy of the final matching process;

[0085] Step S202: determining a third matching set based on the first matching set and the second matching set when the number of intersection features meets a preset set number threshold;

[0086] Exemplarily, that the number of intersection features satisfies a preset set number threshold may be that the number of intersection features is greater than or equal to the preset set number threshold.

[0087] In some embodiments, when the number of intersection features does not meet the preset set number threshold, it means that the number of mutually similar gesture features that meet the requirement is small. In this case, gesture recognition can be directly performed based on the gesture features in the first matching set.

[0088] When there are multiple second matching sets, each second matching set is compared with the first matching set in turn to confirm the number of intersection features. The third matching set is updated according to the comparison results of each second matching set with the first matching set until all second matching sets are traversed.

[0089] Through the above steps S201 and S202, missed recognition of gesture features can be avoided, ensuring that the gesture features in the third matching set are more comprehensive while meeting higher similarity requirements, which is conducive to improving the accuracy of the gesture recognition process.

[0090] For example, the union of the first matching set and the second matching set can be directly used as the third matching set; or, if the number of intersection features meets the preset set number threshold, the union of the first matching set and the second matching set can be used as the third matching set to avoid missed recognition of gesture features during the matching process and improve the recall rate of gesture features. The preset set number threshold can be set according to scenario requirements. For example, the preset set number threshold can be set according to the number of gesture features in the second matching set, for example, the preset set number threshold is set to a ratio of the number of gesture features in the second matching set. The ratio can be set according to the accuracy requirement, for example, 1 / 2 or 1 / 3.

[0091] In some embodiments, determining the first matching set and the second matching set includes: filtering gesture features in a gesture feature library based on similarities between multiple gesture features and matching parameters to obtain the first matching set and the second matching set. Exemplarily, the first matching set is determined by filtering gesture features in the gesture feature library based on the similarity between the gesture feature to be identified and the gesture features in the gesture feature library and a preset first matching parameter, and / or the second matching set is determined by filtering gesture features in the gesture feature library based on the similarity between the target gesture feature and the gesture features in the gesture feature library and a preset second matching parameter. Both the first matching parameter and the second matching parameter are matching parameters used to filter gesture features. Exemplarily, the matching parameters may include multiple parameters, such as a preset similarity threshold, the time at which gesture features were acquired, the number of gesture features in the set, etc. When filtering and determining the set, gesture features with higher similarity may be selected, or each gesture feature may be comprehensively evaluated according to a preset weight to determine whether the gesture feature is included in the corresponding set. In determining the first matching set and the second matching set, filtering based on similarity and preset matching parameters can improve the accuracy of gesture feature matching results.

[0092] In some embodiments, the matching parameter includes a quantity parameter. For example, the first matching parameter includes a first quantity parameter, and the second matching parameter includes a second quantity parameter. During the matching calculation, the second quantity parameter is smaller than the first quantity parameter. Based on the above restrictions, when the similarity requirements between gesture features are met, the number of gesture features in the second matching set is smaller than the number of gesture features in the first matching set, so as to improve the recall rate while improving the accuracy of the gesture feature matching results in the second matching process. For example, when forming the first matching set and the second matching set, the gesture features are first filtered according to the similarity requirements. When the number of gesture features that meet the similarity threshold is greater than the quantity parameter, the gesture features are filtered from high to low according to the similarity, so that the number of gesture features in the first matching set and the second matching set meets the requirements of the quantity parameter.

[0093] Because if two gesture features A and B are similar, then B is in the top k neighbors of A, and conversely, A is also in the top k neighbors of B. But if two gesture features C and D are not similar, even if C is in the top k neighbors of D, D is not in the top k neighbors of C. Therefore, illustratively, the similarity between the gesture feature to be identified and the gesture feature in the gesture feature library can be calculated by the k-reciprocal nearest neighbor (k-rnn) algorithm. Compared with the cosine similarity algorithm that only considers the direction of the vector and ignores the size of the vector, and the shortcomings of a relatively single matching standard, the k-reciprocal nearest neighbor algorithm can effectively improve the recall rate of gesture features while ensuring the recognition accuracy through multiple matching calculations, thereby effectively improving the recognition effect of the current gesture feature and improving the accuracy of gesture matching. Therefore, the first matching set or the second matching set can be calculated by the k-reciprocal nearest neighbor algorithm.

[0094] For example, when similarity is calculated for both the first and second matching sets using the k-nearest-neighbor algorithm, i.e., the similarity between the gesture features to be recognized and the gesture features in the gesture feature library is calculated using the k-nearest-neighbor algorithm, and the similarity between the target gesture features and the gesture features in the gesture feature library is also calculated using the k-nearest-neighbor algorithm, the k value used to calculate the second matching set is smaller than the k value used to calculate the first matching set. For example, if the k value is 20 when calculating the first matching set, the k value can be 10 when calculating the second matching set. Lowering the k value in calculating the second matching set is intended to reduce the amount of computation while ensuring gesture recognition accuracy.

[0095] Since there may be a small number of gesture features in the gesture feature library, or even only one gesture feature that matches the gesture to be recognized, the matching difficulty is relatively high. Therefore, in some embodiments, the gesture feature library can be expanded, and then gesture features are matched based on the expanded gesture feature library. For example, the gesture feature to be recognized is initially matched with the gesture features in the gesture feature library. If the initial match is successful, the gesture feature to be recognized is added to the gesture feature library to obtain the expanded gesture feature library. The initial matching can be achieved using basic similarity calculation methods such as Euclidean distance, cosine distance, and Mahalanobis distance.

[0096] In this embodiment, the initial matching is a rough recognition. The initial matching can determine whether the gesture to be recognized can correspond to the gesture feature in the gesture feature library. If the initial matching is unsuccessful, the subsequent matching process will not be performed, and the gesture feature to be recognized will not be added to the gesture feature library.

[0097] After obtaining the expanded gesture feature library, the gesture feature to be recognized is matched again with the gesture features in the expanded gesture feature library to obtain the first matching set. The matching results at this time include not only the original gesture features in the gesture feature library, but also the gesture feature to be recognized itself.

[0098] After multiple recognitions, the expanded gesture recognition library will contain richer gesture features than the gesture feature library obtained during pre-registration, which is helpful in avoiding missed recognition of gesture features to be recognized.

[0099] For example, since the k-nearest-neighbor algorithm is usually a one-to-many matching process, and the pre-registered gesture feature library has only one gesture feature corresponding to the gesture feature to be recognized in the initial state, it is a one-to-one matching process. Therefore, when this embodiment uses the k-nearest-neighbor algorithm for gesture feature recognition, it can first achieve initial matching through Mahalanobis distance calculation to obtain an expanded gesture feature library.

[0100] In some embodiments, a gesture feature library includes multiple categories of gesture features. In this embodiment, gesture features corresponding to the same hand shape are defined as gesture features of the same type. For example, gesture features corresponding to multiple different OK hand shapes are defined as gesture features of the same type; gesture features corresponding to the OK hand shape and the V hand shape are defined as gesture features of different types. In this embodiment, the gesture feature library includes multiple different types of gesture features. If the number of gesture features of a target type in the gesture feature library exceeds a preset storage limit, the multiple gesture features of the target type in the gesture feature library can be filtered to reduce the data volume and computational complexity of the same type of gesture features, thereby improving the accuracy of gesture feature matching. Different types of gesture features correspond to different preset storage limits, which can be set by the user and modified as needed. For example, the gesture features in the gesture feature library correspond to a variety of functions, such as turning an object on or off, adjusting the volume, and controlling the camera or flash. After expanding the gesture feature library, each category of gesture features has multiple features. If the number of gesture features in a category exceeds its preset storage limit, the gesture features of that category can be filtered. In this embodiment, the target type of gesture feature can be any type of gesture feature in the gesture feature library. During screening, the screening criteria for each type of gesture feature are the preset storage quantity corresponding to that type. The preset storage quantities for different types of gesture features can be the same or different. For example, gesture features corresponding to important functions can have a larger preset storage quantity and a higher similarity requirement. After screening multiple gesture features of the target type in the gesture feature library, an updated gesture feature library is obtained; subsequent target gesture features are then matched against the gesture features in the updated gesture feature library.

[0101] Exemplarily, during the screening process, multiple gesture features of the target type can be screened based on a preset similarity. Exemplarily, the similarity includes a similarity threshold and a similarity magnitude. Gesture features below the similarity threshold can be deleted from the multiple gesture features of the target type, or a preset number of gesture features with a higher similarity can be retained based on the similarity magnitude of each gesture feature. Exemplarily, multiple gesture features of the target type can also be screened based on the acquisition time of each gesture feature in the gesture feature library. Exemplarily, the gesture features are sorted from recent to distant by acquisition time, and a preset number of gesture features with a relatively recent acquisition time are retained. When using similarity and acquisition time to filter gesture features at the same time, you can first use the similarity threshold to filter multiple gesture features of the target type, then set the weights of the similarity size and acquisition time, evaluate the filtered gesture features based on the similarity size, the first weight corresponding to the similarity size, the acquisition time, and the second weight corresponding to the acquisition time, and finally select a preset number of gesture features for storage by sorting them from high to low according to the evaluation value; alternatively, you can first sort the filtered gesture features from high to low according to the similarity size, and then select gesture features in order from near to far according to the acquisition time; alternatively, you can first sort the filtered gesture features from near to far according to the acquisition time, and then select a preset number of gesture features in order from high to low according to the similarity size. It should be noted that during the screening process, the filtered gesture features can also be required to meet at least one of the preset similarity and acquisition time requirements to ensure the accuracy of the gesture feature matching process.

[0102] In some embodiments, FIG3 is a flow chart of a method for obtaining features of a gesture to be recognized according to an embodiment of the present application. As shown in FIG3 , the method includes the following steps:

[0103] Step S301: Acquire the shape features and skeleton point features of the gesture in the image to be detected.

[0104] Step S302: Determine the gesture feature to be recognized based on the appearance feature and the skeleton point feature.

[0105] In this embodiment, the features of the gesture to be recognized are obtained based on both the shape features and the skeleton features in the image to be detected. For example, after obtaining the image to be detected, the shape features and the skeleton features of the gesture are extracted from the image to be detected by a convolutional neural network, and then the shape features and the skeleton features are fused. The fused features are sent to the convolutional neural network for subsequent processing. The input size of the image to be detected is (W, H, C), where W and H represent the width and height of the hand image area, and C represents the number of channels of the hand image. For example, the input size of the hand skeleton point is (1, 21, 3), where 1 is the expanded dimension (corresponding to W), which can be understood as having one hand, 21 is the number of joint points (corresponding to H), and 3 is the (x, y, z) coordinates of the joint points (corresponding to C).

[0106] Exemplarily, an image containing a gesture to be detected is input into the first neural network of a gesture recognition network to obtain the shape features of the gesture. The hand skeletal point data from the image to be detected is input into the second neural network of the gesture recognition network to obtain the skeletal point features. The shape features and skeletal point features are fused and then input into the third neural network of the gesture recognition network to obtain the comprehensive gesture features to be recognized. Feature fusion can be achieved by concatenating the shape features and skeletal point features along the channel dimension. Exemplarily, the third neural network can also adjust the shape features and skeletal point features to have the same data size. To reduce computational complexity, the third neural network can perform dimensionality reduction on the gesture features to be recognized before performing feature extraction.

[0107] Through the above steps S301 and S302, compared with using only the 2D features and angle information of the skeleton points, the solution in this embodiment takes into account both the skeleton point features and the shape features of the gesture, which can effectively avoid misidentification in the retrieval and matching process and improve matching accuracy.

[0108] In the embodiment of the present application, the skeleton point features can be either 3D features or 2.5D features. To improve the detection accuracy of the skeleton point features, the skeleton point features in the embodiment of the present application are illustratively 2.5D features. The (x, y, z) coordinates of the skeleton point, where z represents the depth of the skeleton point relative to the selected root node, such as the wrist point as the root node, reflect the relative depth information between multiple skeleton points, effectively enriching the coordinate information of the skeleton point features.

[0109] In related technologies, gesture recognition technology can recognize gestures that are mostly pre-defined, and users can only use pre-defined gestures, which has poor scalability. In addition, users need to have a certain learning cost when using pre-defined gestures, especially some difficult gestures, which users may not be able to implement, resulting in poor operability.

[0110] The present application also provides a gesture registration method. FIG4 is a flow chart of the gesture registration method according to the present application. The method includes the following steps:

[0111] Step S401, obtaining registration gesture features in the registration image;

[0112] The registration image is an image that includes the user's gesture. The user can upload an existing image as the registration image, or obtain the registration image by real-time capture using the gesture recognition system's image acquisition device. The user can customize the gesture posture according to the scene requirements.

[0113] Step S402: registering the registered gesture feature according to the registered gesture feature and the gesture function corresponding to the registered gesture feature;

[0114] After obtaining the registered gesture features, you can also obtain user-defined gesture functions, such as controlling switches, adjusting volume, etc., and then match the registered gesture features with the corresponding gesture functions to realize gesture registration, so that after the custom gesture is recognized in the future, the corresponding gesture function can be directly executed, thereby optimizing the user's operating experience using gesture recognition.

[0115] Through the above steps S401 and S402, users can register their own custom gestures of interest to the preset gesture feature library, expanding the types of recognizable gestures. Subsequent gestures made by users will be matched with registered gestures and execute corresponding functional instructions.

[0116] Exemplarily, the method for acquiring registered gesture features includes: acquiring shape features and skeleton point features of the gesture in the registered image; and inputting the shape features and skeleton point features into a trained gesture recognition network to acquire registered gesture features.

[0117] For example, when registering a gesture, an identity ID may be assigned to the registered gesture feature to facilitate search and retrieval.

[0118] Based on the above-mentioned gesture recognition method, an embodiment of the present application further provides a network training method for implementing the above-mentioned gesture recognition process. As shown in FIG5 , the method includes:

[0119] Step S501: obtaining a sample pair, which may be one or more of a positive sample pair and a negative sample pair;

[0120] Among them, the positive sample pair includes an anchor sample and a positive sample, and the negative sample pair includes an anchor sample and a negative sample. Each sample contains a gesture image and the skeleton point features corresponding to the gesture image. The gesture image can be obtained by an image acquisition device with a monocular RGB camera;

[0121] Step S502: input the sample pairs into the gesture recognition network to be trained for training.

[0122] In this embodiment, the gesture recognition network is trained using gesture images and hand skeleton point features corresponding to the gestures in the gesture images, so that the comprehensive gesture features obtained by the network integrate the shape features and skeleton point features of the gesture, increase the amount of information of the acquired gesture features, and help improve the accuracy of gesture recognition.

[0123] In some embodiments, training a gesture recognition network based on positive sample pairs includes: inputting an anchor sample and a positive sample in the positive sample pair into the gesture recognition network respectively, and training the gesture recognition network with a loss function value between two gesture comprehensive features output by the two samples through the network being less than a first preset value as a training target, that is, the training target is to shorten the distance between the anchor sample and the positive sample.

[0124] In some embodiments, training a gesture recognition network based on positive sample pairs and negative sample pairs includes inputting two samples in the positive sample pair and two samples in the negative sample pair into the gesture recognition network, and training the gesture recognition network with a loss function value between the difference between two gesture comprehensive features output by the network of the two samples in the positive sample pair and the difference between the gesture comprehensive features output by the network of the two samples in the negative sample pair being greater than a second preset value as a training target, that is, the training target is to shorten the distance between the anchor sample and the positive sample and to increase the distance between the anchor sample and the negative sample.

[0125] In some embodiments, training a gesture recognition network based on negative sample pairs includes: inputting two samples in the negative sample pair into the gesture recognition network respectively, and training the gesture recognition network with a loss function value between two gesture comprehensive features output by the two samples through the network being greater than a third preset value as a training target, that is, the training target is to narrow the distance between the anchor sample and the negative sample.

[0126] For example, the loss function value can be calculated using a soft margin triplet loss function, which is an improvement on the original triplet loss function, for backpropagation. The triplet loss function constructs many sets of triplets (Anchor, Positive, Negative). By learning and optimizing feature vectors, the distance between the anchor sample Anchor and the positive sample Positive in Euclidean space is closer than the distance between the anchor sample Anchor and the negative sample Negative, thereby reducing the intra-class distance and increasing the inter-class distance.

[0127] The triplet loss function can be expressed as: L = max(d(a,p)-d(a,n)+margin,0) (1)

[0128] In formula (1), L represents the loss function, max(*,0) is truncated by setting values ​​less than 0 to 0, a represents the anchor sample; p represents the positive sample, which is a positive sample of the same category as a; n represents the negative sample, which is a negative sample of a different category; margin is a constant greater than 0. The ultimate optimization goal is to shorten the distance between a and p and increase the distance between a and n.

[0129] The soft margin triplet loss function uses the softplus function on the basis of the triplet loss function to replace the truncation function max(*,0) with a smooth approximation, thereby canceling the truncation process. The formula is as follows: s(f)=ln(1+e f ) (2)

[0130] In formula (2), f represents d(a,p)-d(a,n)+margin in formula (1).

[0131] This embodiment uses the soft margin triplet loss function to enhance the generalization ability of the network, and uses the exponentially decaying softplus function to smoothly approximate the truncation function in the triplet loss, thereby improving the training effect of the gesture recognition network.

[0132] The following describes the gesture registration and recognition process through an embodiment.

[0133] The gesture registration and recognition process in this embodiment is implemented by four modules: image acquisition module, target detection and tracking module, hand skeleton point detection module, and feature fusion and extraction module. For example:

[0134] An image acquisition module, configured to drive a monocular RGB camera to implement image acquisition functions, for example, real-time acquisition of human hand images and input to subsequent modules; in other embodiments, an infrared camera or a depth camera may also be used;

[0135] The target detection and tracking module is configured to detect a human hand and match the hand between previous and next frames using a target tracking algorithm. The target detection and tracking module outputs a rectangular frame of the hand in the current frame to the next module. The rectangular frame of the hand in the current frame can be obtained by a target detection model. In addition to obtaining the rectangular frame of the hand, the target detection model can also obtain information such as the hand category, confidence level, and hand ID. The target detection model can be implemented based on deep learning-related theories.

[0136] The hand skeleton point detection module is set to take the hand rectangle output by the target detection and tracking module as input data and output 2.5D skeleton point coordinates;

[0137] The feature fusion and extraction module is configured to receive the hand rectangular image output by the target detection and tracking module and the 2.5D bone point features output by the hand bone point detection module, respectively, to obtain the comprehensive hand features. That is, this module has two inputs.

[0138] During the training phase, the feature fusion and extraction module obtains the comprehensive features of the human hand in the sample image and completes the training of the gesture recognition network through the loss function; during the registration phase, the feature fusion and extraction module obtains the registered gesture features in the registered image as the comprehensive features of the human hand through the trained gesture recognition network, and adds them to the gesture feature library to complete the registration; during the recognition phase, the feature fusion and extraction module collects the current gesture features to be recognized in real time, then compares them with the gesture features in the gesture feature library and outputs the matching results.

[0139] The following is a detailed description of the gesture feature matching process:

[0140] S1, calculate the Mahalanobis distance between the gesture feature to be recognized and each gesture feature in the gesture feature library, compare all Mahalanobis distances with a preset similarity threshold, and obtain an initial matching result; wherein, the calculation formula of the Mahalanobis distance can be expressed as:

[0141] Where x and y represent the gesture features to be identified as the query set probe and the vector of gesture features in the gesture feature library respectively; S is the covariance matrix of the multidimensional random variable;

[0142] S2: If the gesture feature to be identified passes the initial match, the gesture feature to be identified is also saved in the gesture feature library and assigned the same ID as the corresponding gesture feature in the gesture feature library, thereby obtaining an expanded gesture feature library. As shown in FIG6 , the real-time action is the gesture feature to be identified, and the registration set Gallery is the gesture feature library. After the gesture feature to be identified successfully matches the gesture feature initially registered in the Gallery, it is added to the Gallery, thereby obtaining an expanded gesture feature library, i.e., an updated library. The method for determining a successful match includes: the similarity between the gesture feature to be identified and the gesture feature initially registered in the Gallery is greater than or equal to a preset threshold.

[0143] S3, in the expanded gesture feature library, sort the first k gesture features in ascending order according to the Mahalanobis distance between them and the gesture feature to be recognized, and calculate the Jaccard distance between these k gesture features and the gesture feature to be recognized. At this time, the gesture feature corresponding to the smallest Jaccard distance can be used as the matching result of the gesture feature to be recognized; wherein, the calculation formula of the Jaccard distance can be expressed as:

[0144] Where p represents probe; g i Represents the comprehensive features of gestures in the pre-stored gallery; R * It is an extension of R, which represents the set of gesture features that conform to the k-nearest neighbor algorithm and is defined as follows: R(p,k)={(g i ∈N(p,k))∩(p∈N(g i ,k))} (5)

[0145] Where N is the comprehensive feature of gestures from the k-nearest neighbor algorithm, which is defined as follows:

[0146] Where |N(p,k)| represents the number of candidate gesture comprehensive features in N.

[0147] It can be seen that R(p,k) is the set of gesture features in the gesture feature library that have k nearest neighbors with the gesture feature to be recognized when it is used as the probe.

[0148] For example, in order to avoid the exclusion of positive samples from the probe's k-nearest neighbors due to factors such as illumination, occlusion, and angle, the 1 / 2 k-reciprocal nearest neighbors of each candidate in R(p,k) can be incrementally added to a more robust set R according to the following conditions: * (p,k), that is:

[0149] Formula (7) shows that for the gesture feature g in R(p,k) i , traverse these g i , where i = 1, ..., k, and each g i As a new probe, the k-nearest neighbor algorithm with k'=k / 2 is executed in the expanded gesture feature library to obtain the same i Corresponding If R(p,k) and The number of gesture features in the intersection of is greater than or equal to 2 / 3 of the number of gesture features in the The union of R * (p,k).

[0150] Generate R * The process of (p,k) can be referred to Figure 7. In Figure 7, Q is first used as a probe, and the set R(Q,20) is calculated based on the expanded gesture feature library by the k-nearest neighbor algorithm, that is, the first matching set with a k value of 20 is obtained. It should be noted that although the number of pictures contained in the set R is theoretically 20, the gesture features will be filtered during sorting, so the actual number of pictures in the set R may be less than 20; then the picture with ID 2 in R(Q,20) is taken out separately as a new probe, and k is reduced to half of the original, that is, k=20 / 2=10 at this time, and the second row set R(2,10) is calculated again based on the expanded gesture feature library by the k-nearest neighbor algorithm, that is, a second matching set. In the figure, C in R(C,10) represents the ID of the gesture feature, and C is 2 in this embodiment; the number of intersections (i.e., the same gesture features) of R(Q,20) and R(2,10) is 2, and the number of R(2,10) is 3; substitute these numbers into The conditions are met, so we can take the union of R(Q,20) and R(2,10), which is the third row of Figure 7. We traverse all the gesture features in the first row in turn, and finally take the union of all as R * (Q, 20), and a third matching set is obtained; finally, the gesture recognition result can be determined by calculating the similarity between Q and multiple gesture features in the third matching set.

[0151] This embodiment uses the Jaccard distance as a criterion for measuring the similarity between the gesture feature to be recognized and the gesture features in the gesture feature library. From the above theoretical analysis, it can be seen that the use of the Jaccard distance not only considers the similarity between the two gesture features, but also calculates the similarity between the gesture feature to be recognized and a group of similar gesture features, making the information contained in the Jaccard distance richer and eliminating the interference of some environmental factors. It is precisely because of this that this embodiment can significantly improve the effect of gesture recognition by recognizing gestures based on the Jaccard distance. The above matching process based on the k-nearest neighbor algorithm uses the Mahalanobis distance and the Jaccard distance for two matches. Compared with the single matching algorithm in the matching process of the existing gesture recognition technology that only uses the cosine distance as the similarity basis, it can effectively improve the recognition effect of gestures.

[0152] This embodiment also provides a gesture recognition device that is configured to implement the aforementioned embodiments. Details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below, may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0153] FIG8 is a structural block diagram of a gesture recognition device according to an embodiment of the present application. As shown in FIG8 , the device includes a first determination module 81, a second determination module 82, a third determination module 83, and a fourth determination module 84:

[0154] A first determination module 81 is configured to match the gesture feature to be recognized with the gesture features in the gesture feature library to determine a first matching set;

[0155] A second determining module 82 is configured to determine a target gesture feature in the first matching set, match the target gesture feature with gesture features in the gesture feature library, and determine a second matching set;

[0156] A third determining module 83 is configured to determine a third matching set based on the first matching set and the second matching set;

[0157] The fourth determining module 84 is configured to match the gesture feature to be recognized with the gesture features in the third matching set to determine a gesture recognition result of the gesture feature to be recognized.

[0158] Through the above-mentioned gesture recognition device, this embodiment improves the recall rate of the gesture recognition process based on multiple matching processes between the gesture features to be recognized and the gesture features in the gesture feature library, avoids missing recognition of the gesture features to be recognized, and improves the accuracy of gesture recognition.

[0159] Exemplarily, the third determination module 83 is further configured to: determine the number of intersection features based on the intersection of the first matching set and the second matching set; and if the number of intersection features meets a preset set number threshold, determine a third matching set based on the first matching set and the second matching set. Exemplarily, the union of the first matching set and the second matching set can be used as the third matching set.

[0160] Exemplarily, a first matching set is determined based on the similarity between the gesture feature to be identified and the gesture features in the gesture feature library and a preset first matching parameter, and / or a second matching set is determined based on the similarity between the target gesture feature and the gesture features in the gesture feature library and a preset second matching parameter. The first matching parameter includes a first quantity parameter, and the second matching parameter includes a second quantity parameter, where the second quantity parameter is smaller than the first quantity parameter.

[0161] Exemplarily, the similarity between the gesture feature to be recognized and the gesture features in the gesture feature library is calculated using a k-nearest neighbor algorithm.

[0162] Exemplarily, the similarity between the gesture feature to be identified and the gesture feature in the gesture feature library is calculated by a k-nearest neighbor algorithm, and the similarity between the target gesture feature and the gesture feature in the gesture feature library is calculated by a k-nearest neighbor algorithm, and the k value used to calculate the second matching set is smaller than the k value used to calculate the first matching set.

[0163] Exemplarily, matching the gesture features to be identified with the gesture features in the gesture feature library to determine the first matching set includes: initially matching the gesture features to be identified with the gesture features in the gesture feature library, and if the initial matching is successful, adding the gesture features to be identified to the gesture feature library to obtain an expanded gesture feature library; and matching the gesture features to be identified with the gesture features in the expanded gesture feature library again to obtain the first matching set.

[0164] Exemplarily, the gesture feature library includes multiple categories of gesture features. When the number of gesture features of a target type exceeds a preset stored number, the multiple gesture features of the target type are filtered. Exemplarily, filtering the multiple gesture features of the target type includes filtering the multiple gesture features of the target type based on at least one of a preset similarity and an acquisition time of each gesture feature in the gesture feature library.

[0165] Exemplarily, the method for acquiring features of a gesture to be recognized includes: acquiring shape features and skeleton features of the gesture in the image to be detected; and determining features of the gesture to be recognized based on the shape features and the skeleton features. Exemplarily, the skeleton features are 2.5D features.

[0166] Exemplarily, the gesture recognition device also needs to obtain a registration gesture feature in the registration image; and register the registration gesture feature according to the registration gesture feature and the gesture function corresponding to the registration gesture feature.

[0167] An embodiment of the present application also provides a gesture recognition system, including an image acquisition device and a gesture recognition device: the image acquisition device is configured to obtain gesture features to be recognized; the gesture recognition device is configured to match the gesture features to be recognized with gesture features in a gesture feature library to determine a first matching set; determine a target gesture feature in the first matching set, match the target gesture feature with the gesture features in the gesture feature library to determine a second matching set; determine a third matching set based on the first matching set and the second matching set; match the gesture features to be recognized with gesture features in the third matching set to determine a gesture recognition result of the gesture features to be recognized.

[0168] Through the above-mentioned gesture recognition system, this embodiment improves the recall rate of the gesture recognition process based on multiple matching processes between the gesture features to be recognized and the gesture features in the gesture feature library, avoids missed recognition of the gesture features to be recognized, and improves the accuracy of gesture recognition.

[0169] Exemplarily, the gesture recognition device is further configured to: determine a number of intersection features based on the intersection of the first matching set and the second matching set; and, if the number of intersection features meets a preset set number threshold, determine a third matching set based on the first matching set and the second matching set. Exemplarily, the union of the first matching set and the second matching set can be used as the third matching set.

[0170] Exemplarily, a first matching set is determined based on the similarity between the gesture feature to be identified and the gesture features in the gesture feature library and a preset first matching parameter, and / or a second matching set is determined based on the similarity between the target gesture feature and the gesture features in the gesture feature library and a preset second matching parameter. The first matching parameter includes a first quantity parameter, and the second matching parameter includes a second quantity parameter, where the second quantity parameter is smaller than the first quantity parameter.

[0171] Exemplarily, the similarity between the gesture feature to be recognized and the gesture features in the gesture feature library is calculated using a k-nearest neighbor algorithm.

[0172] Exemplarily, the similarity between the gesture feature to be identified and the gesture features in the gesture feature library is calculated by a k-nearest neighbor algorithm, and the similarity between the target gesture feature and the gesture features in the gesture feature library is calculated by a k-nearest neighbor algorithm.

[0173] Exemplarily, matching the gesture features to be identified with the gesture features in the gesture feature library to determine the first matching set includes: initially matching the gesture features to be identified with the gesture features in the gesture feature library, and if the initial matching is successful, adding the gesture features to be identified to the gesture feature library to obtain an expanded gesture feature library; and matching the gesture features to be identified with the gesture features in the expanded gesture feature library again to obtain the first matching set.

[0174] Exemplarily, the gesture feature library includes multiple categories of gesture features. When the number of gesture features of a target type exceeds a preset stored number, the multiple gesture features of the target type are filtered. Exemplarily, filtering the multiple gesture features of the target type includes filtering the multiple gesture features of the target type based on at least one of a preset similarity and an acquisition time of each gesture feature in the gesture feature library.

[0175] Exemplarily, the method for acquiring features of a gesture to be recognized includes: acquiring shape features and skeleton features of the gesture in the image to be detected; and determining features of the gesture to be recognized based on the shape features and the skeleton features. Exemplarily, the skeleton features are 2.5D features.

[0176] Exemplarily, the gesture recognition device also needs to obtain a registration gesture feature in the registration image; and register the registration gesture feature according to the registration gesture feature and the gesture function corresponding to the registration gesture feature.

[0177] An embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the gesture recognition method as described in any of the previous embodiments.

[0178] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A gesture recognition method, the method comprising: Matching the gesture features to be recognized with the gesture features in the gesture feature library to determine a first matching set; Determine a target gesture feature in the first matching set, match the target gesture feature with gesture features in the gesture feature library, and determine a second matching set; Determine a third matching set according to the first matching set and the second matching set; The gesture feature to be recognized is matched with the gesture features in the third matching set to determine a gesture recognition result of the gesture feature to be recognized.

2. The gesture recognition method according to claim 1, further comprising: Determining the number of intersection features according to the intersection of the first matching set and the second matching set; When the number of the intersection features meets a preset set number threshold, the third matching set is determined according to the first matching set and the second matching set.

3. The gesture recognition method according to claim 1 or 2, wherein: Determining a third matching set according to the first matching set and the second matching set includes: The union of the first matching set and the second matching set is used as the third matching set.

4. The gesture recognition method according to claim 1, wherein: Matching the gesture feature to be recognized with the gesture features in the gesture feature library to determine a first matching set includes: Determining the first matching set according to the similarity between the gesture feature to be identified and the gesture feature in the gesture feature library and a preset first matching parameter; and / or, Matching the target gesture feature with the gesture features in the gesture feature library to determine a second matching set includes: The second matching set is determined based on the similarity between the target gesture feature and the gesture features in the gesture feature library and a preset second matching parameter.

5. The gesture recognition method according to claim 4, wherein: The first matching parameter includes a first quantity parameter, and the second matching parameter includes a second quantity parameter, and the second quantity parameter is smaller than the first quantity parameter.

6. The gesture recognition method according to claim 4, wherein: The similarity between the gesture feature to be identified and the gesture features in the gesture feature library is calculated using a k-nearest neighbor algorithm.

7. The gesture recognition method according to claim 6, wherein: The similarity between the target gesture feature and the gesture features in the gesture feature library is calculated by a k-nearest neighbor algorithm, and a k value used to calculate the second matching set is smaller than a k value used to calculate the first matching set.

8. The gesture recognition method according to claim 1, wherein: Matching the gesture feature to be recognized with the gesture features in the gesture feature library to determine a first matching set includes: Initially matching the gesture feature to be identified with the gesture features in the gesture feature library, and if the initial matching is successful, adding the gesture feature to be identified to the gesture feature library to obtain an expanded gesture feature library; The gesture features to be recognized are matched again with the gesture features in the expanded gesture feature library to obtain the first matching set.

9. The gesture recognition method according to claim 1, further comprising: The gesture feature library includes multiple types of gesture features. When the number of gesture features of a target type in the gesture feature library is greater than a preset storage number, the multiple gesture features of the target type are screened.

10. The gesture recognition method according to claim 9, wherein: Screening the multiple gesture features of the target type includes: The multiple gesture features of the target type are screened according to at least one of a preset similarity and an acquisition time of each gesture feature in the gesture feature library.

11. The gesture recognition method according to claim 1, wherein: The method for acquiring the gesture feature to be recognized includes: Obtaining the shape features and skeleton point features of the gesture in the image to be detected; The gesture feature to be recognized is determined based on the appearance feature and the skeleton point feature.

12. The gesture recognition method according to claim 11, wherein: The skeleton point feature is a 2.5D feature.

13. The gesture recognition method according to claim 1, further comprising: Obtaining the registered gesture features in the registered image; According to the registered gesture feature and the gesture function corresponding to the registered gesture feature, the registered gesture feature is registered in the gesture feature library.

14. A gesture recognition device, comprising a first determination module, a second determination module, a third determination module and a fourth determination module: The first determination module is configured to match the gesture feature to be identified with the gesture features in the gesture feature library to determine a first matching set; The second determination module is configured to determine a target gesture feature in the first matching set, match the target gesture feature with gesture features in the gesture feature library, and determine a second matching set; The third determination module is configured to determine a third matching set according to the first matching set and the second matching set; The fourth determination module is configured to match the gesture feature to be recognized with the gesture features in the third matching set to determine a gesture recognition result of the gesture feature to be recognized.

15. A gesture recognition system, comprising an image acquisition device and a gesture recognition device: The image acquisition device is configured to obtain the gesture features to be recognized; The gesture recognition device is configured to match the gesture feature to be recognized with the gesture features in a gesture feature library to determine a first matching set; determine a target gesture feature in the first matching set, match the target gesture feature with the gesture features in the gesture feature library to determine a second matching set; determine a third matching set based on the first matching set and the second matching set; match the gesture feature to be recognized with the gesture features in the third matching set to determine a gesture recognition result of the gesture feature to be recognized.

16. A computer-readable storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the gesture recognition method according to any one of claims 1 to 13.

17. A gesture recognition device, comprising a memory and a processor, wherein the memory stores a program, and when the program is read and executed by the processor, the gesture recognition method according to any one of claims 1 to 13 is implemented.

Images