Network training method and device, gesture recognition method and device, equipment and storage medium

Abstract

The invention provides a network training method and device, a gesture recognition method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring multiple groups of sample gesture data, wherein each group of sample gesture data comprises a sample target gesture image, a positive sample image corresponding to the sample target gesture image, and a negative sample image corresponding to the sample target gesture image; and performing model training according to the multiple groups of sample gesture data to obtain a gesture feature extraction network. Compared with the prior art, the problems that the application scene is limited and the accuracy of the identification result cannot be guaranteed are avoided.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition technology, and in particular to a network training, gesture recognition method, device, equipment and storage medium. Background Art

[0002] Gestures are an important way for humans to express their intentions and are widely used in human-computer interaction (HCI) to transfer information or operate devices. In recent years, with the continuous development of computer vision technology, gesture recognition based on visual data has attracted great interest.

[0003] Compared with traditional gesture recognition methods, gesture recognition based on visual data does not require additional sensors, which makes its application range wider. Gesture recognition based on visual data can capture images or videos of gesture movements through cameras or depth cameras, and identify the type and intention of gestures after processing and analysis by computer vision algorithms. Therefore, gesture recognition based on visual data has been widely used in virtual reality, smart home, smart wearable, games and other fields. Existing gesture recognition work mainly focuses on using various data sets with a limited number of categories to recognize discrete gestures.

[0004] However, this gesture recognition method is not only limited in application scenarios, but also the accuracy of the recognition results cannot be guaranteed. Summary of the invention

[0005] The purpose of the present disclosure is to provide a network training, gesture recognition method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problems in the prior art that the application scenarios are limited and the accuracy of the recognition results cannot be guaranteed.

[0006] To achieve the above objectives, the technical solutions adopted by the embodiments of the present disclosure are as follows:

[0007] In a first aspect, an embodiment of the present disclosure provides a method for training a gesture feature extraction network, the method comprising:

[0008] Acquire multiple groups of sample gesture data, each group of sample gesture data includes: a sample target gesture image, a positive sample image corresponding to the sample target gesture image, and a negative sample image corresponding to the sample target gesture image;

[0009] Model training is performed based on the multiple groups of sample gesture data to obtain a gesture feature extraction network.

[0010] In a second aspect, another embodiment of the present disclosure provides a gesture recognition method, the method comprising:

[0011] Obtaining a gesture image to be recognized;

[0012] Using a pre-trained gesture feature extraction network, extracting features from the gesture image to be identified, and obtaining gesture features of the gesture image to be identified; the gesture feature extraction network is trained according to any one of the methods of the first aspect above;

[0013] According to the gesture feature, a target gesture type of the gesture image to be recognized is determined from a plurality of gesture categories.

[0014] In a third aspect, another embodiment of the present disclosure provides a training device for a gesture feature extraction network, the device comprising:

[0015] The acquisition module is used to acquire multiple groups of sample gesture data, each group of sample gesture data includes: a sample target gesture image, a positive sample image corresponding to the sample target gesture image, and a negative sample image corresponding to the sample target gesture image;

[0016] The determination module is used to perform model training according to the multiple groups of sample gesture data to obtain a gesture feature extraction network.

[0017] In a fourth aspect, another embodiment of the present disclosure provides a gesture recognition device, the device comprising: an acquisition module and a determination module, wherein:

[0018] The acquisition module is used to acquire the gesture image to be recognized;

[0019] The determination module is used to use a pre-trained gesture feature extraction network to extract features from the gesture image to be identified, so as to obtain gesture features of the gesture image to be identified; the gesture feature extraction network is trained according to any one of the methods of the first aspect above; and according to the gesture features, determine the target gesture type of the gesture image to be identified from multiple gesture categories.

[0020] In the fifth aspect, another embodiment of the present disclosure provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the method described in any one of the first aspect or the second aspect above.

[0021] In a sixth aspect, another embodiment of the present disclosure provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspect or the second aspect are executed.

[0022] The beneficial effect of the present disclosure is: by adopting the training method of the gesture feature extraction network provided by the present disclosure, since the multiple groups of sample gesture data in the embodiment of the present disclosure are sample target gesture images obtained after annotation, positive sample images corresponding to the sample target gesture images, and negative sample images corresponding to the sample target gesture images as triplet sample gesture data, the quality of the sample gesture data is high, so the gesture feature extraction network obtained by training the gesture images and annotation information in such multiple groups of triplet sample gesture data can well perceive the subtle differences between different gestures during subsequent use, and the feature information of the gesture image to be identified extracted by such a gesture feature extraction network can reflect the similarity between different gestures, thereby improving the precision and accuracy of subsequent gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present disclosure and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flowchart of a method for training a gesture feature extraction network provided in an embodiment of the present disclosure;

[0025] Figure 2 A flowchart of a method for training a gesture feature extraction network provided by another embodiment of the present disclosure;

[0026] Figure 3 A flowchart of a gesture recognition method provided by an embodiment of the present disclosure;

[0027] Figure 4 A flowchart of a gesture recognition method provided by another embodiment of the present disclosure;

[0028] Figure 5 A schematic diagram of the structure of a training device for a gesture feature extraction network provided in an embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of the structure of a training device for a gesture feature extraction network provided by another embodiment of the present disclosure;

[0030] Figure 7 A schematic diagram of the structure of a gesture recognition device provided by an embodiment of the present disclosure;

[0031] Figure 8 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments.

[0033] The components of the embodiments of the present disclosure generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0034] In addition, the flowchart used in the present disclosure illustrates the operations implemented according to some embodiments of the present disclosure. It should be understood that the operations of the flowchart may not be implemented in order, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart, or may remove one or more operations from the flowchart, under the guidance of the present disclosure.

[0035] The following is an explanation of a method for training a gesture feature extraction network provided by an embodiment of the present disclosure in combination with a number of specific application examples. Figure 1 A flowchart of a method for training a gesture feature extraction network according to an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes:

[0036] S101: Acquire multiple groups of sample gesture data.

[0037] Each set of sample gesture data includes: a sample target gesture image, a positive sample image corresponding to the sample target gesture image, and a negative sample image corresponding to the sample target gesture image.

[0038] For example, in some possible embodiments, for example, the sample target gesture image can be represented by anchor, the positive sample image corresponding to the sample target gesture image can be represented by positive, that is, the positive sample image is a gesture image that is more similar to the anchor, and the negative sample image corresponding to the sample target gesture image can be represented by negative, that is, the image that is less similar to the anchor, so as to construct a subtable marked as<anchor,positive,negative> triplet data (i.e., sample gesture data); wherein, except for the negative sample image, since the target gesture image and the positive sample image are similar images, the target gesture image and the positive sample image can be interchanged.

[0039] In some possible embodiments, the method for determining each group of gesture action data can be, for example: randomly obtaining multiple groups of gesture action data from a gesture action database; performing gesture annotation on the multiple groups of gesture action data to generate multiple groups of sample gesture data; wherein the gesture action database is a preset database, which includes a large number of gesture images, and the source of each gesture image can be, for example, a gesture image obtained on the Internet, or photographed, or pre-downloaded, etc. The source of each gesture image in the specific gesture action database can be flexibly adjusted according to user needs, and is not limited to the above-mentioned embodiments.

[0040] This method of randomly acquiring multiple sets of gesture action data from a gesture action database improves the efficiency of generating multiple sets of gesture action data. Users can directly annotate the generated multiple sets of gesture action data, thereby improving the efficiency of annotation and reducing the waste of time and energy caused by annotation.

[0041] Perform gesture annotation on the gesture action data, that is, annotate the three images in the gesture action data, and annotate each group of gesture action data as<anchor,positive,negative> The triplet data.

[0042] In some possible embodiments, in order to ensure the accuracy and consistency of the annotation, each group of gesture action data can be annotated a preset number of times to obtain an annotation result for each group of gesture action data; based on the annotation results of each group of gesture action data, multiple groups of sample gesture data are generated.

[0043] For example, according to the labeling results of each group of gesture action data, it is determined whether each group of gesture action data is a valid sample group; in each group of gesture action data, it is determined that the valid sample group is a plurality of groups of sample gesture data.

[0044] For example, if the preset number of times is 5 times, during the labeling process, different labelers are required to label the same group of gesture action data, and obtain the labeling results of different labelers for the same group of gesture action data. If the consistency of the labeling results of different labelers for the same group of gesture action data is greater than or equal to the preset threshold, then the group of gesture action data is considered to be a valid sample group; for example, when the preset threshold is 80%, that is, if the labeling results of at least 4 out of 5 labelers are consistent, then the group of gesture action data is determined to be a valid sample group, and multiple valid sample groups together constitute multiple groups of sample gesture data; it should be understood that the above embodiment is only an exemplary description, and the specific preset number of times and the preset threshold settings can be adjusted according to user needs, and are not limited to the above embodiment.

[0045] The multiple sets of sample data obtained in the above manner can reduce the labeling error and thus improve the quality of the multiple sets of sample gesture data. In addition, the multiple sets of sample data obtained in this manner can more comprehensively and completely reflect the differences and similarities of different gesture self-tests, and are suitable for various subsequent gesture action recognition and analysis tasks.

[0046] S102: Perform model training based on multiple groups of sample gesture data to obtain a gesture feature extraction network.

[0047] Since the sample data used for model training in the present disclosure are multiple groups of sample gesture data, each group of sample gesture data includes three gesture images, and the gesture images are all labeled. This setting makes the quality of the sample gesture data high. Therefore, the gesture feature extraction network trained with the gesture images and annotation information in such multiple groups of sample gesture data can well perceive the subtle differences between different gestures during use. The feature information of the gesture image to be identified is extracted through such a gesture feature extraction network. The feature information can reflect the similarity between different gestures and improve the precision and accuracy of subsequent gesture recognition.

[0048] By adopting the training method of the gesture feature extraction network provided in the present disclosure, since the multiple groups of sample gesture data in the embodiments of the present disclosure are triplet sample gesture data obtained after annotation, the quality of the sample gesture data is high. Therefore, the gesture feature extraction network obtained by training the gesture images and annotation information in such multiple groups of triplet sample gesture data can well perceive the subtle differences between different gestures during subsequent use. The feature information of the gesture image to be identified extracted by such a gesture feature extraction network can reflect the similarity between different gestures, thereby improving the precision and accuracy of subsequent gesture recognition.

[0049] Optionally, based on the above embodiment, the embodiment of the present disclosure may further provide a method for training a gesture feature extraction network. The following is an example of the implementation process of obtaining the gesture feature extraction network in the above method in combination with the accompanying drawings. Figure 2 A flowchart of a gesture feature extraction method provided by another embodiment of the present disclosure is shown in FIG. Figure 2 As shown, S102 may include:

[0050] S111: Using a preset initial gesture feature extraction network to perform feature extraction on multiple groups of sample gesture data to obtain sample gesture features of the multiple groups of sample gesture data.

[0051] Among them, the sample gesture features of each group of sample gesture data respectively include: the gesture features of the target gesture image, the gesture features of the positive sample image corresponding to the target gesture image, and the gesture features of the negative sample image corresponding to the target gesture image. The gesture features of the above three images together constitute the sample gesture features of this group of sample gesture data.

[0052] S112: Calculate the loss function value of each group of sample gesture data according to the sample gesture features of the multiple groups of sample gesture data.

[0053] In some possible embodiments, the method for determining the loss function value can be, for example, determining the loss function value of each group of sample gesture data based on the distance between the positive sample image and the target gesture image, the distance between the negative sample image and the target gesture image in each group of sample gesture data, and a preset threshold.

[0054] During training, multiple sets of sample gesture data are<A,P,N> Input into the initial gesture feature extraction network, and get<f(A),f(P),f(N)> ; Wherein, A is the anchor in a set of sample gesture data, that is, the target gesture image; P is the positive in a set of sample gesture data, that is, the positive sample image corresponding to the sample target gesture image; N is the negative in a set of sample gesture data, that is, the negative sample image corresponding to the sample target gesture image; f(A) is the gesture feature of A, f(P) is the gesture feature of P, and f(N) is the gesture feature of N. According to the above three features of each set of sample gesture data, the loss function of each set of sample gesture data is determined respectively.

[0055] For example, in the embodiment of the present disclosure, the loss function L triplet The determination method of can be obtained by the following formula, for example:

[0056] L triplet (A,,)=max(‖f(A)-(P)‖ 2 -‖f(A)-(N)‖ 2 +,0)

[0057] Where, ‖f(A)-(P)‖ 2 It represents the distance between the sample target gesture image and the positive sample image, ‖f(A)-(N)‖ 2 It represents the distance between the sample target gesture image and the negative sample image, α,0 represents the preset tolerance threshold, where α is a value greater than 0, L triplet (,P,N) needs to be greater than 0; that is, during the model training process, it is necessary to use the loss function L triplet (A, P, N) continuously iterates the model. During the iteration process, it is necessary to ensure that L tripletThe value of (A,,) is greater than 0.

[0058] S113: According to the loss function value, the model parameters of the initial gesture feature extraction network are adjusted until a preset iteration stop condition is reached.

[0059] In the embodiment of the present disclosure, if L triplet If the value of (A,,) is greater than or equal to a preset threshold, a contrastive learning method is used to shorten the distance between the sample target gesture image and the positive sample image in the group of sample gesture data, and to increase the distance between the sample target gesture image and the negative sample image, so as to obtain the adjusted sample gesture features corresponding to the group of sample gesture data, and re-determine the loss function value corresponding to the group of sample gesture data according to the adjusted sample gesture features, until the loss function value corresponding to the group of sample gesture data is less than the preset threshold and greater than 0.

[0060] If L ttiplet If the value of (,P,N) is less than the preset threshold and greater than 0, it means that the loss function corresponding to the group of sample gesture data has converged, then other groups of sample gesture data in the multiple groups of sample gesture data are obtained, and the loss function values ​​of the other groups of sample gesture data are determined; finally, when the loss function value of each group of sample gesture data in the multiple groups of sample gesture data is less than the preset threshold and greater than 0, it is determined that the model has converged, and the converged network at this time is determined to be a gesture feature extraction network.

[0061] During the training process, since the annotations of the sample gesture data are all annotations for the gestures, no attention is paid to other contents of each image in the sample gesture data. Therefore, the gesture feature extraction network trained in this way only pays attention to the gesture posture in the gesture image in the subsequent process of recognizing the gesture image, and only obtains the gesture features for the gesture, which has nothing to do with factors such as image lighting and background in the gesture image.

[0062] Since the training process of the gesture feature extraction network provided by the present invention is based on a plurality of pre-labeled triplet sample data for training, the trained gesture feature extraction network can characterize the gesture posture of any gesture and can reflect the similarity and difference between different gesture postures. The gesture feature extracted by the gesture feature extraction network trained by the present invention is a continuous gesture feature characterization method, which can significantly improve the perception accuracy and application scope of gestures. That is, the gesture feature extraction network provided by the present invention can be applied to any scenario requiring gesture recognition. The trained gesture feature extraction network is applied in different gesture recognition scenarios without the need for re-collection and re-training of gesture data, and can be directly applied to different gesture recognition scenarios. That is, the gesture feature extraction network trained by the training method of the gesture feature extraction network provided by the present invention has a wide application range and strong applicability.

[0063] By adopting the gesture feature extraction network provided by the present invention, by pre-labeling multiple triplet sample gesture data and using the contrastive learning method to train the gesture feature extraction network, a gesture feature extraction network with good generalization ability and robustness can be obtained. The gesture feature extraction network can not only convert different gesture images into representation vectors (gesture features) with corresponding semantics, but also provide support for subsequent gesture recognition tasks.

[0064] Optionally, based on the above embodiment, the embodiment of the present disclosure may further provide a gesture recognition method. The implementation process of the above method is illustrated below with reference to the accompanying drawings. Figure 3 A flowchart of a gesture recognition method provided by another embodiment of the present disclosure is shown in FIG. 1 . The method may include:

[0065] S201: Acquire a gesture image to be recognized.

[0066] In some possible embodiments, the gesture image to be recognized may be obtained by, for example, a photographed gesture image, or a gesture image selected from a photo album, or a gesture image input by other means. The specific gesture image input method is not limited to this and can be flexibly adjusted according to user needs. The gesture image to be recognized needs to include complete hand content and clear gesture content.

[0067] S202: Using a pre-trained gesture feature extraction network, extract features from the gesture image to be recognized, and obtain gesture features of the gesture image to be recognized.

[0068] The gesture feature extraction network is based on the above Figure 1-Figure 2 trained by any method.

[0069] Since the gesture feature extraction network labels the sample gesture data with annotations for gestures during the training process, the gesture feature extraction network trained in this way only focuses on the gesture posture in the gesture image to be identified during the process of identifying the gesture image, and only obtains gesture features for the gesture posture, which is unrelated to factors such as image lighting and background in the gesture image to be identified.

[0070] S203: Determine a target gesture type of the gesture image to be recognized from multiple gesture categories according to the gesture feature.

[0071] In some possible embodiments, multiple gesture categories may include, for example, a heart gesture, an OK gesture, a number four gesture, a five-finger open gesture, a fist gesture, a scissors hand gesture, or a thumbs-up gesture, etc. The gesture content that may be included in the specific multiple preset gesture categories can be flexibly adjusted according to user needs and is not limited to those given in the above embodiments.

[0072] Since the gesture feature extraction network provided by the present invention is trained based on multiple groups of triplet sample data with annotations, the gesture features provided by the present invention can be used to well identify gestures in different scenarios, so that the application scenarios of the gesture feature extraction network provided by the present invention are wide, and can be applied to any scenario where gesture recognition is required. Compared with the prior art, there is no need to perform time-consuming and labor-intensive data collection work for different scenarios, but only needs to preset multiple gesture categories according to different usage scenarios, and then directly use the trained gesture feature extraction network to extract the gesture features of the gesture image to be identified, and then directly determine the target gesture category from multiple gesture categories based on the identified gesture features. Since the features extracted by the gesture feature extraction network provided by the present invention can more finely represent the gesture posture of the gesture image to be identified, the target gesture category identified by the gesture recognition method provided by the application is accurately determined.

[0073] With the gesture recognition method provided by the present invention, since the gesture feature extraction network uses multiple groups of labeled triplet sample gesture data during the training process, such a training method enables the gesture feature extraction network to have strong expressiveness for gesture posture when extracting features. When determining the target gesture category based on the extracted gesture features, the gesture feature extraction network can perceive the subtle features of the gesture, thereby improving the perception ability of the gesture features, and determining the gesture type with high accuracy. Moreover, the network can be applied in any scenario where gesture recognition is required, and has wide applicability.

[0074] Optionally, based on the above embodiment, the embodiment of the present disclosure may further provide a gesture recognition method. The implementation process of determining the target gesture type in the above method is illustrated below with reference to the accompanying drawings. Figure 4 A flowchart of a gesture recognition method provided by another embodiment of the present disclosure is shown as follows: Figure 4 As shown, S203 may include:

[0075] S211: Determine the similarity between the gesture feature and the preset gesture features of each gesture category.

[0076] In some possible examples, each gesture category includes multiple gesture examples. For example, each gesture category includes m gesture examples corresponding to the gesture category. For example, if the number of preset gesture categories is L, then in the process of classifying the gesture features of the gesture image to be recognized, the given L preset gesture categories are marked as {C 1 , C 2 ,…,C l ,…,C L}, where l represents the serial number of the category, C l represents the gesture category of the lth category. Each gesture category C l There are m gesture examples These examples describe the gesture posture of the target, regardless of factors such as image lighting and background. In order to classify the gesture image x to be identified, first use the trained hand gesture feature extraction network G to map the image x to the universal gesture representation space, obtain the gesture features corresponding to the gesture image x to be identified, and then calculate the average distance d between the gesture features of the gesture image x to be identified and the gesture vectors (gesture features) of each gesture example under each gesture category. l , based on the average distance, determine the similarity between the gesture features of the gesture image x to be recognized and each gesture category.

[0077]

[0078] In the above method of determining the similarity between the gesture feature of the gesture image x to be recognized and each gesture category, since the similarity is jointly determined based on multiple gesture instances under each gesture category, the similarity result obtained is more accurate.

[0079] S212: Determine the gesture category with the smallest similarity as the target gesture category.

[0080] Among them, due to the above distance d lIt represents the similarity between the gesture image x to be recognized and each gesture category. The smaller the average distance between the gesture image x to be recognized and the gesture category A, the higher the similarity between the gesture image x to be recognized and the gesture category A; the larger the average distance between the gesture image x to be recognized and the gesture category B, the lower the similarity between the gesture image x to be recognized and the gesture category B; incident for the gesture image x to be recognized, the gesture category with the smallest average distance from the gesture image x to be recognized can be determined, which is the final gesture recognition result of the gesture image x to be recognized. The result can be expressed as argmin l ( l (x)).

[0081] With the gesture recognition method provided by the present invention, since the gesture feature extraction network uses multiple groups of labeled triplet sample gesture data during the training process, such a training method enables the gesture feature extraction network to have strong expressive ability for gesture posture when extracting features. When determining the gesture posture based on the extracted gesture features, the gesture feature extraction network can perceive the subtle features of the gesture, thereby improving the perception ability of the gesture features. Moreover, when determining the gesture type of the gesture image to be identified, there are multiple gesture examples under each gesture category. By determining the distance between the gesture to be identified and the multiple gesture examples and taking the average value, the target gesture category to which the gesture to be identified belongs is determined. In this way, the accuracy of determining the target gesture category is high, and the network can be applied to any scenario where gestures need to be identified, thus having wide applicability.

[0082] The following is an explanation of the training device of the gesture feature extraction network provided by the present disclosure in conjunction with the accompanying drawings. The training device of the gesture feature extraction network can perform the above Figure 1-Figure 2 The specific implementation and beneficial effects of the training method of any gesture feature extraction network are as mentioned above and will not be described in detail below.

[0083] Figure 5 A schematic diagram of the structure of a training device for a gesture feature extraction network provided in an embodiment of the present disclosure, such as Figure 5 As shown, the device includes: an acquisition module 301 and a determination module 302, wherein:

[0084] An acquisition module 301 is used to acquire multiple groups of sample gesture data, each group of sample gesture data includes: a sample target gesture image, a positive sample image corresponding to the sample target gesture image, and a negative sample image corresponding to the sample target gesture image;

[0085] The determination module 302 is used to perform model training according to multiple groups of sample gesture data to obtain a gesture feature extraction network.

[0086] Optionally, based on the above embodiment, the embodiment of the present disclosure may also provide a training device for a gesture feature extraction network. Figure 5 The implementation process of the given device is described by way of example. Figure 6 A schematic diagram of a training device for a gesture feature extraction network according to another embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the device also includes: a marking module 303, wherein:

[0087] An acquisition module 301 is specifically used to randomly acquire multiple groups of gesture action data from a gesture action database;

[0088] The labeling module 303 is used to label multiple groups of gesture action data to generate multiple groups of sample gesture data.

[0089] Optionally, the labeling module 303 is specifically used to label each group of gesture action data for a preset number of times to obtain a labeling result for each group of gesture action data; and generate multiple groups of sample gesture data according to the labeling result of each group of gesture action data.

[0090] Optionally, the determination module 303 is specifically used to determine whether each group of gesture action data is a valid sample group according to the labeling results of each group of gesture action data; in each group of gesture action data, determine that the valid sample group is a plurality of groups of sample gesture data.

[0091] Optionally, the determination module 303 is specifically used to use a preset initial gesture feature extraction network to perform feature extraction on multiple groups of sample gesture data to obtain sample gesture features of the multiple groups of sample gesture data; calculate the loss function value of each group of sample gesture data based on the sample gesture features of the multiple groups of sample gesture data; and adjust the model parameters of the initial gesture feature extraction network based on the loss function value until a preset iteration stop condition is reached.

[0092] Optionally, the determination module 303 is specifically used to determine the loss function value of each group of sample gesture data based on the distance between the positive sample image and the target gesture image, the distance between the negative sample image and the target gesture image in each group of sample gesture data, and a preset threshold.

[0093] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0094] The following is an explanation of the gesture recognition device provided by the present disclosure in conjunction with the accompanying drawings. The gesture recognition device can perform the above Figure 3-Figure 4 For any gesture recognition method, its specific implementation and beneficial effects are as mentioned above and will not be described in detail below.

[0095] Figure 7A schematic diagram of the structure of a gesture recognition device provided by an embodiment of the present disclosure is shown in FIG. Figure 7 As shown, the device includes: an acquisition module 401 and a determination module 402, wherein:

[0096] An acquisition module 401 is used to acquire a gesture image to be recognized;

[0097] The determination module 402 is used to use a pre-trained gesture feature extraction network to extract features of the gesture image to be identified, and obtain gesture features of the gesture image to be identified; the gesture feature extraction network is based on the above Figure 1-Figure 2 The method is obtained by training by any one of the methods; according to the gesture features, the target gesture type of the gesture image to be recognized is determined in multiple gesture categories.

[0098] Optionally, the determination module 402 is specifically configured to determine the similarity between the gesture feature and preset gesture features of each gesture category; and determine the gesture category with the smallest similarity as the target gesture category.

[0099] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0100] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0101] Figure 8 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. The electronic device can be integrated into a terminal device or a chip of the terminal device.

[0102] like Figure 8 As shown, the electronic device includes: a processor 501 , a bus 502 and a storage medium 503 .

[0103] The processor 501 is used to store programs. The processor 501 calls the program stored in the storage medium 503 to execute the above Figure 1-Figure 4The corresponding method embodiment has similar specific implementation methods and technical effects, which will not be described here.

[0104] Optionally, the present disclosure further provides a program product, such as a storage medium, on which a computer program is stored, including a program, which, when executed by a processor, executes an embodiment corresponding to the above method.

[0105] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0108] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

Claims

1. A training method for a gesture feature extraction network, characterized in that: include: Acquire multiple groups of sample gesture data, each group of sample gesture data includes: a sample target gesture image, a positive sample image corresponding to the sample target gesture image, and a negative sample image corresponding to the sample target gesture image; Model training is performed based on the multiple groups of sample gesture data to obtain a gesture feature extraction network.

2. The method according to claim 1, characterized in that The obtaining of multiple sets of sample gesture data includes: Randomly obtain multiple groups of gesture action data from a gesture action database; Perform gesture annotation on the multiple sets of gesture action data to generate the multiple sets of sample gesture data.

3. The method according to claim 2, characterized in that The step of performing gesture annotation on the plurality of groups of gesture action data to generate the plurality of groups of sample gesture data includes: Performing gesture annotation for a preset number of times on each group of gesture action data to obtain annotation results for each group of gesture action data; The multiple groups of sample gesture data are generated according to the labeling results of the respective groups of gesture action data.

4. The method according to claim 3, characterized in that The generating the plurality of groups of sample gesture data according to the labeling results of the groups of gesture action data comprises: Determining whether each group of gesture action data is a valid sample group according to the labeling results of each group of gesture action data; Among the groups of gesture action data, a valid sample group is determined to be the multiple groups of sample gesture data.

5. The method according to claim 1, characterized in that The model training is performed according to the multiple groups of sample gesture data to obtain a gesture feature extraction network, including: Using a preset initial gesture feature extraction network to perform feature extraction on the multiple groups of sample gesture data to obtain sample gesture features of the multiple groups of sample gesture data; Calculating a loss function value of each group of sample gesture data according to the sample gesture features of the multiple groups of sample gesture data; According to the loss function value, the model parameters of the initial gesture feature extraction network are adjusted until a preset iteration stop condition is reached.

6. The method according to claim 5, characterized in that The loss function value of each group of sample gesture data is calculated according to the sample gesture features of the multiple groups of sample gesture data, including: The loss function value of each group of sample gesture data is determined according to the distance between the positive sample image and the target gesture image, the distance between the negative sample image and the target gesture image in each group of sample gesture data, and a preset threshold.

7. A gesture recognition method, characterized in that: The method comprises: Obtaining a gesture image to be recognized; Using a pre-trained gesture feature extraction network, extracting features from the gesture image to be identified, and obtaining gesture features of the gesture image to be identified; the gesture feature extraction network is trained according to any one of the methods of claims 1 to 6 above; According to the gesture feature, a target gesture type of the gesture image to be recognized is determined from a plurality of gesture categories.

8. The method according to claim 7, characterized in that The step of determining the target gesture type of the gesture image to be recognized from a plurality of gesture categories according to the gesture feature includes: Determining the similarity between the gesture feature and the preset gesture features of each of the gesture categories; The gesture category with the smallest similarity is determined as the target gesture category.

9. A training device for a gesture feature extraction network, characterized in that: The device comprises: an acquisition module and a determination module, wherein: The acquisition module is used to acquire multiple groups of sample gesture data, each group of sample gesture data includes: a sample target gesture image, a positive sample image corresponding to the sample target gesture image, and a negative sample image corresponding to the sample target gesture image; The determination module is used to perform model training according to the multiple groups of sample gesture data to obtain a gesture feature extraction network.

10. A gesture recognition device, characterized in that: The device comprises: an acquisition module and a determination module, wherein: The acquisition module is used to acquire the gesture image to be recognized; The determination module is used to use a pre-trained gesture feature extraction network to extract features from the gesture image to be identified, so as to obtain gesture features of the gesture image to be identified; the gesture feature extraction network is trained according to any one of the methods of claims 1 to 6 above; and according to the gesture features, determine the target gesture type of the gesture image to be identified in multiple gesture categories.

11. An electronic device, characterized in that: The device includes: a processor, a storage medium and a bus, the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1 to 8.

12. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images