Gesture recognition method and device, electronic equipment and storage medium

Abstract

The invention provides a gesture recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a gesture image, extracting the gesture features of the gesture image, and enabling the gesture features to serve as the shared features of a plurality of subtask branches for achieving gesture recognition; the plurality of subtask branches at least comprise a gesture category classification branch and a palm orientation classification branch; processing the gesture features through the gesture category classification branch to generate first sub-attribute information representing gesture categories; processing the gesture features through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation; and based on the first sub-attribute information and the second sub-attribute information, judging whether the gesture image contains a gesture representing a specified action or not. According to the technical scheme provided by the invention, the accuracy of gesture recognition can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a gesture recognition method, device, electronic device and storage medium. Background Art

[0002] In some application scenarios, it is necessary to recognize the user's gestures and determine whether to execute a certain command based on the recognition results.

[0003] However, gestures usually vary greatly, and the hands may also hold objects. Such complexity in real-world applications increases the difficulty of gesture recognition.

[0004] In view of this, an accurate gesture recognition solution is currently needed. Summary of the invention

[0005] In view of this, one or more embodiments of the present disclosure provide a gesture recognition method, device, electronic device, and storage medium, which can improve the accuracy of gesture recognition.

[0006] On the one hand, the present disclosure provides a gesture recognition method, the method comprising: acquiring a gesture image, and extracting gesture features of the gesture image, the gesture features serving as shared features of multiple subtask branches for implementing gesture recognition; the multiple subtask branches at least include a gesture category classification branch and a palm orientation classification branch; processing the gesture features through the gesture category classification branch to generate first sub-attribute information representing the gesture category; processing the gesture features through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation; and judging whether the gesture image contains a gesture representing a specified action based on the first sub-attribute information and the second sub-attribute information.

[0007] On the other hand, the present disclosure also provides a gesture recognition device, which includes: a feature extraction unit, used to obtain a gesture image and extract gesture features of the gesture image, wherein the gesture features serve as shared features of multiple subtask branches for implementing gesture recognition; the multiple subtask branches include at least a gesture category classification branch and a palm orientation classification branch; a first processing unit, used to process the gesture features through the gesture category classification branch to generate first sub-attribute information representing the gesture category; a second processing unit, used to process the gesture features through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation; and a gesture judgment unit, used to judge whether the gesture image contains a gesture representing a specified action based on the first sub-attribute information and the second sub-attribute information.

[0008] Another aspect of the present disclosure provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned gesture recognition method is implemented.

[0009] On the other hand, the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned gesture recognition method is implemented.

[0010] The technical solution provided by one or more embodiments of the present disclosure can analyze gesture features from different dimensions through multiple subtask branches when identifying whether a gesture image contains a gesture representing a specified action. Specifically, the subtask branch may include at least a gesture category classification branch and a palm orientation classification branch. Through different subtask branches, different attribute information of gesture features can be obtained. By combining these attribute information, it can be more accurately determined whether the gesture image contains a gesture representing a specified action. In addition, by sharing gesture features through different subtask branches, the model architecture can be simplified and the use of multiple independent classification models can be avoided, thereby improving the efficiency of gesture recognition while ensuring the accuracy of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The features and advantages of the embodiments of the present disclosure may be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0012] Figure 1 The figure shows the overall structure of a gesture recognition model in one embodiment of the present disclosure;

[0013] Figure 2 A schematic diagram showing the steps of a gesture recognition method in one embodiment of the present disclosure is shown;

[0014] Figure 3 A processing logic diagram of a refine module in one embodiment of the present disclosure is shown;

[0015] Figure 4 A schematic diagram of dividing a training sample set in one embodiment of the present disclosure is shown;

[0016] Figure 5 A schematic diagram of functional modules of a gesture recognition device in one embodiment of the present disclosure is shown;

[0017] Figure 6 A schematic structural diagram of an electronic device in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0019] In one embodiment of the present disclosure, a gesture recognition model is provided. Figure 1 , the gesture recognition model may include a shared feature extraction network and multiple different subtask branches. Each subtask branch may include a refinement module (refine) and a multilayer perceptron (MLP). The refinement modules in different subtask branches may perform differentiated processing on the gesture features output by the feature extraction network, and the processed results may output the attribute information of the corresponding subtask branch through the multilayer perceptron. The attribute information output by different subtask branches may describe the gesture features in the gesture image from different dimensions. Subsequently, by combining the attribute information output by each subtask branch, it may be more accurately determined whether the gesture image contains a gesture representing a specified action.

[0020] See also Figure 2 An embodiment of the present disclosure provides a gesture recognition method, which may include the following steps.

[0021] S1: Acquire a gesture image and extract gesture features of the gesture image, wherein the gesture features serve as shared features of multiple subtask branches for implementing gesture recognition; the multiple subtask branches include at least a gesture category classification branch and a palm orientation classification branch.

[0022] In this embodiment, the gesture features of the gesture image can be Figure 1 The feature extraction network shown in the figure is extracted. In practical applications, the feature extraction network can be flexibly set according to actual needs. For example, a network such as VGGNet or ResNet can be used to implement it.

[0023] Generally speaking, the gesture features extracted by the feature extraction network are often downsampled to a certain extent compared to the original gesture image. For example, the resolution of the original gesture image is H*W, where H represents the height of the gesture image and W represents the width of the gesture image. Then the resolution of the extracted gesture features can be (H / 32)*(W / 32).

[0024] In addition, the output result of the feature extraction network may contain multiple channels, each channel can output a processing result with a resolution of (H / 32)*(W / 32). In this way, the processing results output by each channel are summarized. Assuming that the total number of channels is C, the extracted gesture feature can be a three-dimensional feature such as (H / 32, W / 32, C).

[0025] In this embodiment, the designated action to be identified can be an action of opening the hand and the palm facing the image acquisition device. Then, in the gesture image containing the designated action, the palm should be unfolded, and the center of the palm should be captured in the gesture image. In view of this, after the gesture features are extracted, multiple subtask branches can be used to process the gesture features separately, so as to obtain the attribute information output by each subtask branch. Among them, the multiple subtask branches can at least include a gesture category classification branch and a palm direction classification branch. The gesture category classification branch can identify whether the palm in the gesture image is unfolded, and the palm direction classification branch can determine whether the center of the palm can be captured in the gesture image. Of course, depending on the application scenario, more subtask branches can be included to describe the gesture features based on different dimensions and improve the accuracy of gesture recognition.

[0026] In the related art, if you want to implement a multi-task recognition process, you can usually establish different recognition models. For example, you can separately establish a gesture category recognition model and a palm orientation recognition model to handle their respective recognition tasks. However, considering the performance of the actual application device, if multiple recognition models are deployed in the application device, it will often consume a lot of resources of the application device and cannot meet the lightweight requirements. Different recognition models often have different requirements for data processing, which also makes the fusion process of the recognition model difficult.

[0027] In view of this, in this embodiment, the gesture features extracted from the gesture image can be used as shared features of each subtask branch. Figure 1 As shown in the figure, multiple subtask branches share the same feature extraction network, thus avoiding repeated extraction of gesture features; in addition, each subtask branch can refine features through its own refine module, and output recognition results based on its own MLP module, thus reflecting the differentiation of different attribute information. In this way, it can avoid redundant feature extraction processes and obtain the output results of different subtask branches, while improving data processing efficiency and ensuring the diversity of output results, thus providing a basis for accurate gesture recognition.

[0028] S3: Processing the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category.

[0029] S5: Processing the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation.

[0030] In this embodiment, the gesture features extracted by the feature extraction network can be used as shared features for the subsequent gesture category classification branch and the palm orientation classification branch. The gesture category classification branch and the palm orientation classification branch can both be used to implement classification tasks, but different classification tasks may have different processing processes for gesture features.

[0031] Specifically, in the relevant classification process, the extracted image features can be directly pooled to obtain pooled features without changing the number of channels. However, each channel in the pooled feature usually corresponds to only one one-dimensional value, and directly compressing the two-dimensional image features into a one-dimensional value will lose more feature information.

[0032] In view of this, in order to ensure the accuracy of gesture recognition, in this embodiment, when the gesture features are processed by the refine module in the gesture category classification branch, the extracted gesture features are not directly pooled, but Figure 3 As shown in the figure, while keeping the feature dimension in each channel unchanged, the number of channels of the gesture feature is first expanded. In this way, the feature dimension in each channel can remain unchanged. Since the number of channels is expanded, the feature dimension finally obtained will also increase. Specifically, in order to keep the feature dimension in the channel unchanged, a 1*1 convolution kernel (conv1*1) can be used to process the gesture feature. By setting the number of filter components (filters), the number of channels after expansion can be determined. For example, in Figure 3 In the example, conv1*1 can be used to double the number of channels of the gesture feature, thereby obtaining a gesture feature with a dimension of (H / 32, W / 32, 2C), where C can represent the original number of channels of the gesture feature extracted by the feature extraction network.

[0033] In this embodiment, after the channel expansion process, the gesture features with the expanded number of channels can be obtained, and the gesture features with the expanded number of channels can be subsequently reshaped in their respective channels to generate channel reshaped features. For example, in the gesture features after the number of channels is expanded, the gesture features in any channel can be represented as a two-dimensional feature such as (H / 32, W / 32). By reshaping the two-dimensional feature, a one-dimensional feature with a length of H / 32*W / 32 can be obtained. It should be noted that the feature values ​​at each position in the two-dimensional feature are usually not changed during the reshaping process. It is just that the feature values ​​originally arranged in a two-dimensional manner are converted to be arranged in sequence in a one-dimensional manner. For example, the original two-dimensional feature can be represented by the following 3*3 matrix:

[0034]

[0035] Then after reshaping the two-dimensional feature, we can get the following one-dimensional feature with a length of 9:

[0036] (A,B,C,D,E,F,G,H,J)

[0037] In this way, the two-dimensional features in each channel can be reshaped into one-dimensional features. By integrating the one-dimensional features in each channel, a channel reshaped feature that can reflect the number of channels can be formed. Since the channel reshaped feature introduces the number of channels, it can be represented by a two-dimensional feature. For example, the gesture feature after the channel number expansion can be represented as a three-dimensional feature (H / 32, W / 32, 2C), and the generated channel reshaped feature can be represented as a two-dimensional feature (H / 32*W / 32, 2C).

[0038] After the above-mentioned processing of keeping the feature dimension unchanged (or called size unchanged) and expanding the number of channels, the feature connection between each channel can be deepened, so that the processed gesture features are richer. Compared with direct pooling, this processing method in this embodiment can retain more information of gesture features, and at the same time can also mine the deep features between each channel of gesture features, providing a reliable data basis for subsequent gesture recognition.

[0039] In this embodiment, after generating the channel reshaping feature, in order to reduce the amount of data processing, the channel reshaping feature can be compressed to obtain a one-dimensional feature whose length is the number of expanded channels. The one-dimensional feature can then be input into a multi-layer perceptron for processing and the multi-layer perceptron outputs the first sub-attribute information representing the gesture category.

[0040] Specifically, assuming that the dimension of the channel reshaping feature is (H / 32*W / 32, 2C), the feature in each channel can be compressed into a feature value, and finally a one-dimensional feature with a length of 2C can be obtained. The multi-layer perceptron can generate the first sub-attribute information representing the gesture category based on the one-dimensional feature by processing the one-dimensional feature.

[0041] In practical applications, a multilayer perceptron may include a fully connected layer and an activation layer. After the fully connected layer processes the one-dimensional feature, a logits (corrected value) of length M may be obtained, where M may represent the number of categories of a preset gesture category. Subsequently, the logits of length M may be processed by the sigmoid function or the softmax function in the activation layer to obtain a probability distribution vector of the one-dimensional feature in the gesture category classification task. Each probability value in the probability distribution vector may correspond one-to-one to a gesture category. In this way, by determining the classification result of the gesture category represented by the probability distribution vector, the first sub-attribute information representing the gesture category may be generated based on the classification result. For example, the maximum probability value may be determined from the probability distribution vector, and the gesture category corresponding to the maximum probability value may be used as the classification result of the gesture category represented by the probability distribution vector. The classification result may be used to generate the first sub-attribute information representing the gesture category.

[0042] For example, in the scenario of detecting whether the gesture is an open palm, the corresponding gesture categories may be "open palm" and "closed palm". Of course, some scenarios may also include categories such as "unrecognizable". In this way, the gesture recognition process can be regarded as a binary classification process (assuming that the classification results are only "open palm" and "closed palm"). If the probability value representing the category "open palm" in the probability distribution vector is the highest, then the first sub-attribute information representing the gesture category "open palm" can be output. The first sub-attribute information can be, for example, a value of "1", while the first sub-attribute information representing the gesture category "closed palm" can be a value of "0". Of course, the above explanations of gesture categories and attribute information are only examples in specific scenarios. Those skilled in the art can flexibly interpret the specific meanings of gesture categories and attribute information according to actual scenario requirements, and these explanations should be included in the protection scope of this disclosure.

[0043] Similarly, the process of processing the gesture features through the palm orientation classification branch to generate the second sub-attribute information representing the palm orientation can also be done through Figure 3The model architecture shown in the figure is used for processing. However, in the data processing process of the palm direction classification branch, the required data complexity can be lower than that of the gesture category classification branch. Therefore, when processed by conv1*1, the number of channels can be expanded without expansion or with a lower multiple. For example, assuming that the number of channels in the gesture category classification branch is expanded to 4 times, the number of channels in the palm direction classification branch can be expanded to 2 times, or the number of channels can be not expanded. The specific method can be flexibly set according to the needs of the actual application scenario.

[0044] In one embodiment, the refine module in the palm orientation classification branch may only perform convolution processing on the gesture features with unchanged feature dimensions. After the convolution processing, the gesture features may also undergo a channel reshaping and compression process. Specifically, the gesture features after the convolution processing may be reshaped in their respective channels to generate channel reshaped features. The channel reshaped features may then be feature compressed to obtain a one-dimensional feature having a length equal to the number of channels after expansion. The one-dimensional feature may be processed by the MLP module in the palm orientation classification branch, thereby generating second sub-attribute information characterizing the palm orientation based on the one-dimensional feature.

[0045] The specific process of channel reshaping and feature compression in the above implementation can be found in the corresponding description in the previous implementation, and will not be repeated here.

[0046] Similarly, the multi-layer perceptron in the palm orientation classification branch may include a fully connected layer and an activation layer. After the fully connected layer processes the one-dimensional feature, a logits (corrected value) of length N can be obtained, where N can represent the number of pre-set palm orientation classifications. Subsequently, the logits of length N can be processed by the sigmoid function or the softmax function in the activation layer to obtain the probability distribution vector of the one-dimensional feature in the palm orientation classification task. By determining the classification result of the palm orientation represented by the probability distribution vector, the second sub-attribute information representing the palm orientation can be generated based on the classification result. The specific process can also be found in the description in the aforementioned embodiment, which will not be repeated here.

[0047] S7: Based on the first sub-attribute information and the second sub-attribute information, determine whether the gesture image includes a gesture representing a designated action.

[0048] In this embodiment, after obtaining the first sub-attribute information and the second sub-attribute information through different sub-task branches, it is possible to determine whether the gesture image contains a gesture representing a specified action based on a combination of the first sub-attribute information and the second sub-attribute information.

[0049] For example, the above-mentioned designated action can be an action of opening the hand with the palm facing the image acquisition device. If the gesture category represented by the first sub-attribute information is an open hand, and the palm direction represented by the second sub-attribute information is a palm facing the image acquisition device, then it can be determined that the gesture of the designated action can be identified from the gesture image. After the gesture of the designated action is identified, the trigger instruction bound to the gesture can be executed. For example, the instruction can be automatic photo taking. Then, when the automatic photo taking function is integrated into the image acquisition device, when this function is activated, the image acquisition device will capture the image in front of the lens. Once the gesture of "open hand and palm facing the image acquisition device" is detected in the image, a photo can be taken automatically.

[0050] From the perspective of data processing, the first sub-attribute information and the second sub-attribute information can usually be represented by numerical values. For example, when the first sub-attribute information is 1, it indicates that the hand is open, and when it is 0, it indicates that the hand is not open. Similarly, when the second sub-attribute information is 1, it indicates that the palm is facing the image acquisition device (the palm can be captured), and when it is 0, it indicates that the palm is not facing the image acquisition device. Then, only when the combination information of (1, 1) is obtained, it is considered that the gesture of the specified action is recognized from the gesture image.

[0051] In one embodiment, in addition to analyzing the gesture features to obtain the gesture category and palm orientation, the left and right hands can be further identified. Specifically, the above-mentioned multiple subtask branches can also include a left and right hand classification branch, and the left and right hand classification branch can still include a refine module and an MLP module. By processing the shared gesture features through the left and right hand classification branches, it can be identified whether the gesture features correspond to the left hand or the right hand.

[0052] In one embodiment, the gesture features can be processed by the left and right hand classification branch to generate third sub-attribute information characterizing the left and right hand categories. The left and right hand classification branch can be regarded as performing a binary classification task. Of course, in addition to the left hand category and the right hand category, if other categories (such as the "unrecognizable" category) are included in the classification task, then the left and right hand classification branch can also be regarded as performing a multi-classification task, and the present disclosure does not limit this.

[0053] Subsequently, by combining the sub-attribute information output by each of the three sub-task branches, it is possible to determine whether the gesture image contains a gesture representing a specified action based on the first sub-attribute information, the second sub-attribute information, and the third sub-attribute information. The specific judgment idea is similar to the description in the aforementioned implementation method, and will not be repeated here.

[0054] In one embodiment, when the gesture features are processed by the left and right hand classification branch, since the data processing demand is also lower than that of the gesture category classification branch, after the gesture features are subjected to convolution processing with unchanged feature dimensions, the number of channels may not be expanded or the number of channels may be expanded at a lower rate. Taking the example of not expanding the number of channels, the gesture features after convolution processing may be reshaped in their respective channels to generate channel reshaped features. Then, the channel reshaped features may be feature compressed in a manner similar to the above to obtain a one-dimensional feature with a length equal to the number of channels after expansion, and based on the one-dimensional feature, the third sub-attribute information characterizing the left and right hand categories may be generated.

[0055] In actual application scenarios, the above-mentioned feature extraction network and each subtask branch can be pre-trained by machine learning. Taking the gesture category classification branch as an example, a training sample set of the branch can be obtained, which can include multiple image samples with gesture categories marked. Figure 1 and Figure 3 The model in is processed to generate the prediction result in the gesture category classification task. There may be an error between the prediction result and the gesture category pre-labeled by the image sample, and the error can be calculated by the cross-entropy loss function (Cross-Entropy Loss).

[0056] For different subtask branches, a similar method can be used to calculate the errors of each branch. Subsequently, by weighted summing the errors of each subtask branch, the comprehensive error of the overall model can be obtained, which can be used to correct the overall model composed of the feature extraction network and each subtask branch, thereby realizing the sample training process.

[0057] In one embodiment, in order to further improve the accuracy of gesture recognition, for the training sample set applicable to the gesture category classification branch, the image samples representing the open hand can be further subdivided into two refined sample sets: the palm facing the image acquisition device and the back of the hand facing the image acquisition device. Figure 4, a first sample set representing an open hand and a second sample set representing a closed hand can be obtained from the training sample set according to the pre-labeled gesture category labels. For the first sample set, a first subset containing palm features can be obtained, and a second subset not containing palm features can be obtained. The inclusion of palm features can be understood as the ability to detect the palm from the image sample, indicating that the palm is facing the image acquisition device at this time; and the non-inclusion of palm features can be understood as the inability to detect the palm from the image sample, in which case the back of the hand may be facing the image acquisition device, or although the palm is facing the image acquisition device, the hand is grasping other objects, resulting in the palm being blocked. This situation will still be classified as not containing palm features. Generally speaking, the detection of palm features can be achieved through manual labeling.

[0058] After obtaining the first subset and the second subset, the image samples in the first subset can be labeled with a first label, the image samples in the second subset can be labeled with a second label, and the image samples in the second sample set can be labeled with a third label, and the first label, the second label and the third label are different from each other. In this way, although they all represent the situation of an open hand, by detecting the palm feature, they can be further refined into two different labels, the first label and the second label, and for the situation where the hand is not open, it can be labeled with a third label that is different from both.

[0059] Subsequently, the model is trained by using image samples annotated with three types of labels, and the resulting gesture recognition model can accurately identify situations where the hand is open and the palm is facing the image acquisition device. It can also accurately identify situations where the hand is open and the palm is facing the image acquisition device, but the palm is blocked by other objects held in the hand. This can improve the gesture recognition accuracy of the model and can be applied to complex application scenarios, thereby avoiding commands being triggered incorrectly. For example, under normal circumstances, the automatic photo-taking function will only be triggered when it is detected that the hand is open and the palm is facing the image acquisition device, and the palm features cannot be blocked. The technical solution provided by the present disclosure can accurately distinguish various situations and improve the accuracy of command execution.

[0060] It should be noted that, although the gesture category classification branch can also preliminarily identify whether the palm is facing the image acquisition device by refining the training samples, if the recognition result is inconsistent with the recognition result of the palm direction classification branch, the recognition result of the palm direction classification branch can usually be used. Of course, in practical applications, certain weights can also be used for selection, which can be flexibly determined according to the application scenario.

[0061] The technical solution provided by one or more embodiments of the present disclosure can analyze gesture features from different dimensions through multiple subtask branches when identifying whether a gesture image contains a gesture representing a specified action. Specifically, the subtask branch may include at least a gesture category classification branch and a palm orientation classification branch. Through different subtask branches, different attribute information of gesture features can be obtained. By combining these attribute information, it can be more accurately determined whether the gesture image contains a gesture representing a specified action. In addition, by sharing gesture features through different subtask branches, the model architecture can be simplified and the use of multiple independent classification models can be avoided, thereby improving the efficiency of gesture recognition while ensuring the accuracy of gesture recognition.

[0062] In specific application scenarios, by segmenting the training samples, it is also possible to accurately identify whether the palm features are blocked, thereby ensuring the correct triggering of the command and reducing the false touch rate of the command.

[0063] See also Figure 5 In one embodiment of the present disclosure, a gesture recognition device is provided, the device comprising:

[0064] A feature extraction unit 100, configured to obtain a gesture image and extract gesture features of the gesture image, wherein the gesture features are used as shared features of multiple subtask branches for implementing gesture recognition; the multiple subtask branches at least include a gesture category classification branch and a palm orientation classification branch;

[0065] A first processing unit 200, configured to process the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category;

[0066] A second processing unit 300, configured to process the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation;

[0067] The gesture determination unit 400 is configured to determine whether the gesture image includes a gesture representing a designated action based on the first sub-attribute information and the second sub-attribute information.

[0068] Among them, the specific processing logic of each functional module can be found in the description of the aforementioned method implementation method, which will not be repeated here.

[0069] Each unit described in the above embodiments may be implemented by a computer chip or a product having a certain function. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0070] For the convenience of description, the above devices are described in terms of functions and are described separately in various units. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0071] See also Figure 6 An embodiment of the present disclosure further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned gesture recognition method is implemented.

[0072] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned gesture recognition method is implemented.

[0073] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0074] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0075] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0076] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0077] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0078] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0079] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A gesture recognition method, characterized in that: The method comprises: Acquire a gesture image, and extract gesture features of the gesture image, wherein the gesture features are used as shared features of multiple subtask branches for implementing gesture recognition; the multiple subtask branches include at least a gesture category classification branch and a palm orientation classification branch; Processing the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category; Processing the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation; Based on the first sub-attribute information and the second sub-attribute information, it is determined whether the gesture image includes a gesture representing a designated action.

2. The method according to claim 1, characterized in that Processing the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category includes: Expanding the number of channels of the gesture features, and reshaping the gesture features after the number of channels has been expanded in their respective channels to generate channel reshaping features; Compressing the channel reshaping feature to obtain a one-dimensional feature with a length equal to the number of expanded channels; Based on the one-dimensional feature, first sub-attribute information representing the gesture category is generated.

3. The method according to claim 2, characterized in that Based on the one-dimensional feature, generating first sub-attribute information representing the gesture category includes: Calculating the probability distribution vector of the one-dimensional feature in the gesture category classification task; A classification result of the gesture category represented by the probability distribution vector is determined, and first sub-attribute information representing the gesture category is generated based on the classification result.

4. The method according to claim 1, characterized in that: The method further comprises: For a training sample set applicable to the gesture category classification branch, obtaining a first sample set representing an open hand and a second sample set representing a closed hand from the training sample set; In the first sample set, a first subset including palm features is obtained, and a second subset not including palm features is obtained; Annotate the image samples in the first subset with a first label, annotate the image samples in the second subset with a second label, and annotate the image samples in the second sample set with a third label; The first label, the second label and the third label are different from each other.

5. The method according to claim 1, characterized in that: Processing the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation includes: Performing convolution processing on the gesture features with unchanged feature dimensions, and reshaping the gesture features after the convolution processing in their respective channels to generate channel reshaping features; Compressing the channel reshaping feature to obtain a one-dimensional feature with a length equal to the number of expanded channels; Based on the one-dimensional feature, second sub-attribute information representing the direction of the palm is generated.

6. The method according to claim 5, characterized in that Generating second sub-attribute information representing the palm orientation based on the one-dimensional feature includes: Calculating a probability distribution vector of the one-dimensional feature in a palm orientation classification task; A classification result of the palm orientation represented by the probability distribution vector is determined, and second sub-attribute information representing the palm orientation is generated based on the classification result.

7. The method according to claim 1, characterized in that The plurality of subtask branches also include a left-hand classification branch; the method further includes: Processing the gesture feature through the left-right hand classification branch to generate third sub-attribute information representing the left-right hand category; Based on the first sub-attribute information, the second sub-attribute information, and the third sub-attribute information, it is determined whether the gesture image includes a gesture representing a designated action.

8. The method according to claim 7, characterized in that Processing the gesture feature by the left-right hand classification branch to generate third sub-attribute information representing the left-right hand category includes: Performing convolution processing on the gesture features with unchanged feature dimensions, and reshaping the gesture features after the convolution processing in their respective channels to generate channel reshaping features; Compressing the channel reshaping feature to obtain a one-dimensional feature with a length equal to the number of expanded channels; Based on the one-dimensional feature, third sub-attribute information characterizing the left and right hand categories is generated.

9. A gesture recognition device, characterized in that: The device comprises: A feature extraction unit, used to obtain a gesture image and extract gesture features of the gesture image, wherein the gesture features are used as shared features of multiple subtask branches for implementing gesture recognition; the multiple subtask branches at least include a gesture category classification branch and a palm orientation classification branch; A first processing unit, configured to process the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category; A second processing unit, configured to process the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation; A gesture determination unit is configured to determine whether the gesture image contains a gesture representing a designated action based on the first sub-attribute information and the second sub-attribute information.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images