Gesture recognition method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a gesture recognition method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring a gesture image, and extracting gesture features of the gesture image, the gesture features being used as shared features of a plurality of sub-task branches for implementing gesture recognition, and the plurality of sub-task branches at least comprising a gesture category classification branch and a palm orientation classification branch; processing the gesture features by means of the gesture category classification branch so as to generate first sub-attribute information representing a gesture category; processing the gesture features by means of the palm orientation classification branch so as to generate second sub-attribute information representing a palm orientation; and, on the basis of the first sub-attribute information and the second sub-attribute information, determining whether the gesture image comprises a gesture representing a specified action.

Description

A gesture recognition method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 28, 2023, with application number 202311278579.3 and invention name “A gesture recognition method, device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of image processing technology, and in particular to a gesture recognition method, device, electronic device, and storage medium. Background Art

[0003] In some application scenarios, it is necessary to recognize the user's gestures and determine whether to execute a certain instruction based on the recognition results.

[0004] However, gestures usually vary greatly, and the hands may also hold objects. This complexity in real-world applications increases the difficulty of gesture recognition.

[0005] In view of this, an accurate gesture recognition solution is currently needed.

[0006] Summary of the Invention

[0007] In view of this, one or more embodiments of the present disclosure provide a gesture recognition method, apparatus, electronic device, and storage medium, which can improve the accuracy of gesture recognition.

[0008] On one hand, the present disclosure provides a gesture recognition method, the method comprising: acquiring a gesture image and extracting gesture features of the gesture image, the gesture features serving as shared features of multiple subtask branches for implementing gesture recognition; the multiple subtask branches including at least a gesture category classification branch and a palm orientation classification branch; processing the gesture features by the gesture category classification branch to generate first sub-attribute information representing the gesture category; processing the gesture features by the palm orientation classification branch to generate second sub-attribute information representing the palm orientation; and determining whether the gesture image contains a gesture representing a specified action based on the first sub-attribute information and the second sub-attribute information.

[0009] On the other hand, the present disclosure also provides a gesture recognition device, which includes: a feature extraction unit, which is used to obtain a gesture image and extract gesture features of the gesture image, wherein the gesture features serve as shared features of multiple subtask branches for implementing gesture recognition; the multiple subtask branches include at least a gesture category classification branch and a palm orientation classification branch; a first processing unit, which is used to process the gesture features through the gesture category classification branch to generate first sub-attribute information representing the gesture category; a second processing unit, which is used to process the gesture features through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation; and a gesture judgment unit, which is used to judge whether the gesture image contains a gesture representing a specified action based on the first sub-attribute information and the second sub-attribute information.

[0010] Another aspect of the present disclosure provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program. When the computer program is executed by the processor, the above-mentioned gesture recognition method is implemented.

[0011] Another aspect of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned gesture recognition method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The features and advantages of the various embodiments of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0013] FIG1 shows a schematic diagram of the overall structure of a gesture recognition model in one embodiment of the present disclosure;

[0014] FIG2 is a schematic diagram showing steps of a gesture recognition method in one embodiment of the present disclosure;

[0015] FIG3 shows a schematic diagram of the processing logic of a refine module in one embodiment of the present disclosure;

[0016] FIG4 shows a schematic diagram of dividing a training sample set in one embodiment of the present disclosure;

[0017] FIG5 is a schematic diagram showing the functional modules of a gesture recognition device according to an embodiment of the present disclosure;

[0018] FIG6 shows a schematic structural diagram of an electronic device in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present disclosure.

[0020] In one embodiment of the present disclosure, a gesture recognition model is provided, please refer to Figure 1, and the gesture recognition model may include a shared feature extraction network and multiple different subtask branches. Each subtask branch may include a refinement module (refine) and a multi-layer perceptron (MLP). The refinement modules in different subtask branches can perform differentiated processing on the gesture features output by the feature extraction network, and the processed results can output the attribute information of the corresponding subtask branch through the multi-layer perceptron. The attribute information output by different subtask branches can describe the gesture features in the gesture image from different dimensions. Subsequently, combined with the attribute information output by each subtask branch, it can be more accurately judged whether the gesture image contains a gesture representing a specified action.

[0021] Referring to FIG. 2 , an embodiment of the present disclosure provides a gesture recognition method, which may include the following steps.

[0022] S1: Acquire a gesture image and extract gesture features of the gesture image. The gesture features serve as shared features of multiple subtask branches for implementing gesture recognition. The multiple subtask branches include at least a gesture category classification branch and a palm orientation classification branch.

[0023] In this embodiment, the gesture features of the gesture image can be extracted using the feature extraction network shown in FIG1 . In practical applications, the feature extraction network can be flexibly configured according to actual needs. For example, a network such as VGGNet or ResNet can be used for implementation.

[0024] Generally speaking, gesture features extracted by the feature extraction network are often downsampled to a certain extent compared to the original gesture image. For example, if the resolution of the original gesture image is H*W, where H represents the height of the gesture image and W represents the width of the gesture image, the resolution of the extracted gesture features can be (H / 32)*(W / 32).

[0025] In addition, the output results of the feature extraction network may contain multiple channels, and each channel can output a processing result with a resolution of (H / 32)*(W / 32). In this way, by summarizing the processing results output by each channel, assuming that the total number of channels is C, the extracted gesture feature can be a three-dimensional feature such as (H / 32, W / 32, C).

[0026] In this embodiment, the specified action to be identified can be an action of opening the hand with the palm facing the image acquisition device. Then, in the gesture image containing the specified action, the palm should be unfolded, and the center of the palm should be captured in the gesture image. In view of this, after the gesture features are extracted, multiple subtask branches can be used to process the gesture features separately, so as to obtain the attribute information output by each subtask branch. Among them, the multiple subtask branches can at least include a gesture category classification branch and a palm direction classification branch. The gesture category classification branch can identify whether the palm in the gesture image is unfolded, and the palm direction classification branch can determine whether the center of the palm can be captured in the gesture image. Of course, depending on the application scenario, more subtask branches can be included to describe the gesture features based on different dimensions and improve the accuracy of gesture recognition.

[0027] In related technologies, if you want to implement a multi-task recognition process, you can usually establish different recognition models. For example, you can establish a gesture category recognition model and a palm orientation recognition model to handle their respective recognition tasks. However, considering the performance of actual application devices, if multiple recognition models are deployed in an application device, it will often consume a large amount of application device resources and fail to meet the lightweight requirements. Different recognition models often have different data processing requirements, which also makes the recognition model fusion process difficult.

[0028] In view of this, in this embodiment, the gesture features extracted from the gesture image can be used as shared features for each subtask branch. As shown in Figure 1, multiple subtask branches share the same feature extraction network, thereby avoiding repeated extraction of gesture features. In addition, each subtask branch can refine the features through its own refine module and output the recognition results based on its own MLP module, thereby reflecting the differentiation of different attribute information. In this way, redundant feature extraction processes can be avoided while obtaining the output results of different subtask branches. While improving data processing efficiency, the diversity of output results is guaranteed, thus providing a basis for accurate gesture recognition.

[0029] S3: Processing the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category.

[0030] S5: Processing the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation.

[0031] In this embodiment, the gesture features extracted by the feature extraction network can serve as shared features for the subsequent gesture category classification branch and palm orientation classification branch. Both the gesture category classification branch and the palm orientation classification branch can be used to implement classification tasks, but different classification tasks may require different processing of gesture features.

[0032] Specifically, during the relevant classification process, the extracted image features can be directly pooled, thereby obtaining pooled features without changing the number of channels. However, each channel in the pooled features usually only corresponds to a one-dimensional value. Directly compressing the two-dimensional image features into a one-dimensional value will lose a lot of feature information.

[0033] In view of this, in order to ensure the accuracy of gesture recognition, in this embodiment, when the gesture features are processed by the refine module in the gesture category classification branch, the extracted gesture features are not directly pooled. Instead, as shown in FIG3 , the number of channels of the gesture features is first expanded while keeping the feature dimension in each channel unchanged. In this way, the feature dimension in each channel can remain unchanged. Since the number of channels is expanded, the feature dimension finally obtained will also increase. Specifically, in order to keep the feature dimension in the channel unchanged, a 1*1 convolution kernel (conv1*1) can be used to process the gesture features, and the number of channels after expansion can be determined by setting the number of filter components (filters). For example, in FIG3 , conv1*1 can be used to double the number of channels of the gesture feature, thereby obtaining a gesture feature with a dimension of (H / 32, W / 32, 2C), where C can represent the original number of channels of the gesture feature extracted by the feature extraction network.

[0034] In this embodiment, after the channel expansion process, the gesture features with expanded channel numbers can be obtained, and the gesture features with expanded channel numbers can be subsequently reshaped in their respective channels to generate channel reshaped features. For example, in the gesture features after the channel number expansion, the gesture features in any channel can be represented as a two-dimensional feature such as (H / 32, W / 32). By reshaping the two-dimensional feature, a one-dimensional feature with a length of H / 32*W / 32 can be obtained. It should be noted that the feature values ​​at each position in the two-dimensional feature are usually not changed during the reshaping process. It is just that the feature values ​​originally arranged in a two-dimensional manner are converted to be arranged in sequence in a one-dimensional manner. For example, the original two-dimensional feature can be represented by the following 3*3 matrix:

[0035] Then after reshaping the two-dimensional feature, we can get the following one-dimensional feature with a length of 9:

[0036] (A,B,C,D,E,F,G,H,J)

[0037] In this way, the two-dimensional features in each channel can be reshaped into one-dimensional features. By integrating the one-dimensional features in each channel, a channel-reshaped feature that reflects the number of channels can be constructed. Since the channel-reshaped feature introduces the number of channels, it can be represented by a two-dimensional feature. For example, the gesture feature after the channel number expansion can be represented as a three-dimensional feature (H / 32, W / 32, 2C), and the resulting channel-reshaped feature can be represented as a two-dimensional feature (H / 32*W / 32, 2C).

[0038] The aforementioned processing of maintaining feature dimensions (or size) and expanding the number of channels deepens the feature connections between channels, enriching the processed gesture features. Compared to direct pooling, this processing method in this embodiment can retain more information about gesture features while also exploring the deep features between gesture feature channels, providing a reliable data foundation for subsequent gesture recognition.

[0039] In this embodiment, after generating the channel reshaping feature, in order to reduce the amount of data processing, the channel reshaping feature can be compressed to obtain a one-dimensional feature with a length equal to the number of expanded channels. The one-dimensional feature can then be input into a multi-layer perceptron for processing, and the multi-layer perceptron outputs the first sub-attribute information representing the gesture category.

[0040] Specifically, assuming the dimension of the channel reshaped feature is (H / 32*W / 32, 2C), the features in each channel can be compressed into a single feature value, ultimately obtaining a one-dimensional feature of length 2C. The multilayer perceptron processes this one-dimensional feature and generates the first sub-attribute information representing the gesture category based on it.

[0041] In practical applications, a multi-layer perceptron may include a fully connected layer and an activation layer. After the fully connected layer processes the one-dimensional feature, a logits (corrected value) of length M may be obtained, where M may represent the number of categories of a preset gesture category. Subsequently, the logits of length M may be processed by the sigmoid function or the softmax function in the activation layer to obtain a probability distribution vector of the one-dimensional feature in the gesture category classification task. Each probability value in the probability distribution vector may correspond one-to-one to a gesture category. In this way, by determining the classification result of the gesture category represented by the probability distribution vector, the first sub-attribute information representing the gesture category may be generated based on the classification result. For example, the maximum probability value may be determined from the probability distribution vector, and the gesture category corresponding to the maximum probability value may be used as the classification result of the gesture category represented by the probability distribution vector. The classification result may be used to generate the first sub-attribute information representing the gesture category.

[0042] For example, in the scenario of detecting whether the gesture is an open palm, the corresponding gesture categories may be "open palm" and "closed palm". Of course, some scenarios may also include categories such as "unrecognizable". In this way, the gesture recognition process can be regarded as a binary classification process (assuming that the classification results are only "open palm" and "closed palm"). If the probability value representing the category "open palm" in the probability distribution vector is the highest, then the first sub-attribute information representing the gesture category "open palm" can be output. The first sub-attribute information can be, for example, a value of "1", while the first sub-attribute information representing the gesture category "closed palm" can be a value of "0". Of course, the above explanations of gesture categories and attribute information are only examples in specific scenarios. Those skilled in the art can flexibly interpret the specific meanings of gesture categories and attribute information according to the actual scenario requirements, and these explanations should be included in the scope of protection of this disclosure.

[0043] Similarly, the process of processing gesture features through the palm orientation classification branch to generate the second sub-attribute information representing the palm orientation can also be processed by the model architecture shown in Figure 3. However, in the data processing process of the palm orientation classification branch, the required data complexity can be lower than that of the gesture category classification branch. Therefore, when processed by conv1*1, the number of channels can be expanded without expansion or with a lower multiple. For example, assuming that the number of channels in the gesture category classification branch is expanded to 4 times, the number of channels in the palm orientation classification branch can be expanded to 2 times, or the number of channels can be not expanded. The specific method can be flexibly set according to the needs of the actual application scenario.

[0044] In one embodiment, the refine module in the palm orientation classification branch can only perform convolution processing on the gesture features with unchanged feature dimensions. After the convolution processing, the gesture features can also undergo channel reshaping and compression processing. Specifically, the gesture features after convolution processing can be reshaped in their respective channels to generate channel reshaping features. The channel reshaping features can then be feature compressed to obtain a one-dimensional feature with a length equal to the number of channels after expansion. This one-dimensional feature can be processed by the MLP module in the palm orientation classification branch, thereby generating second sub-attribute information representing the palm orientation based on the one-dimensional feature.

[0045] The specific process of channel reshaping and feature compression in the above embodiment can be found in the corresponding description in the above embodiment, which will not be repeated here.

[0046] Similarly, the multi-layer perceptron in the palm orientation classification branch may include a fully connected layer and an activation layer. After the fully connected layer processes the one-dimensional feature, a logits (corrected value) of length N may be obtained, where N may represent the number of pre-set palm orientation classifications. Subsequently, the logits of length N may be processed by the sigmoid function or the softmax function in the activation layer to obtain a probability distribution vector of the one-dimensional feature in the palm orientation classification task. By determining the classification result of the palm orientation represented by the probability distribution vector, the second sub-attribute information representing the palm orientation may be generated based on the classification result. The specific process may also be referred to the description in the aforementioned embodiment, which will not be repeated here.

[0047] S7: Based on the first sub-attribute information and the second sub-attribute information, determine whether the gesture image includes a gesture representing a designated action.

[0048] In this embodiment, after obtaining the first sub-attribute information and the second sub-attribute information through different sub-task branches, it is possible to determine whether the gesture image contains a gesture representing a specified action based on the combination of the first sub-attribute information and the second sub-attribute information.

[0049] For example, the above-mentioned designated action can be an action of opening the hand with the palm facing the image acquisition device. If the gesture category represented by the first sub-attribute information is an open hand, and the palm direction represented by the second sub-attribute information is a palm facing the image acquisition device, then it can be determined that the gesture of the designated action can be identified from the gesture image. After the gesture of the designated action is identified, the trigger instruction bound to the gesture can be executed. For example, the instruction can be to take a photo automatically. Then, when the automatic photo taking function is integrated into the image acquisition device, when this function is activated, the image acquisition device will capture the image in front of the lens. Once the gesture of "open hand and palm facing the image acquisition device" is detected in the image, a photo can be taken automatically.

[0050] From a data processing perspective, the first and second sub-attribute information can typically be represented by numerical values. For example, a value of 1 for the first sub-attribute indicates an open hand, while a value of 0 indicates a closed hand. Similarly, a value of 1 for the second sub-attribute indicates the palm is facing the image capture device (capable of capturing the palm), while a value of 0 indicates the palm is not facing the image capture device. Therefore, only when the combination of (1, 1) is obtained is the gesture image considered to have recognized the specified action.

[0051] In one embodiment, in addition to analyzing gesture features to determine gesture categories and palm orientations, further identification of left and right hands can be performed. Specifically, the multiple subtask branches described above can also include a left-hand classification branch, which can still include a refine module and an MLP module. By processing the shared gesture features through the left-hand classification branch, it can be determined whether the gesture features correspond to the left or right hand.

[0052] In one embodiment, the left-hand and right-hand classification branches can process the gesture features to generate third sub-attribute information representing the left-hand and right-hand categories. The left-hand and right-hand classification branches can be considered to perform a binary classification task. Of course, if the classification task includes other categories (such as an "unrecognizable" category) in addition to the left-hand and right-hand categories, then the left-hand and right-hand classification branches can also be considered to perform a multi-classification task, which is not limited in this disclosure.

[0053] Subsequently, by combining the sub-attribute information output by each of the three subtask branches, it is possible to determine whether the gesture image contains a gesture representing a specified action based on the first, second, and third sub-attribute information. The specific approach to this determination is similar to that described in the previous embodiment and will not be repeated here.

[0054] In one embodiment, when the gesture features are processed by the left and right hand classification branch, the data processing requirements are also lower than those of the gesture category classification branch. Therefore, after the gesture features are subjected to convolution processing with the feature dimension unchanged, the number of channels may not be expanded or the number of channels may be expanded at a lower rate. Taking the example of not expanding the number of channels, the gesture features after convolution processing can be reshaped in their respective channels to generate channel reshaped features. Then, the channel reshaped features can be feature compressed in a manner similar to the above to obtain a one-dimensional feature with a length equal to the number of channels after expansion, and based on the one-dimensional feature, the third sub-attribute information representing the left and right hand categories is generated.

[0055] In actual application scenarios, the above-mentioned feature extraction network and each subtask branch can be pre-trained through machine learning. Taking the gesture category classification branch as an example, a training sample set for the branch can be obtained, which can include multiple image samples with gesture categories labeled. Subsequently, these image samples can be processed by the models in Figures 1 and 3 to generate prediction results in the gesture category classification task. There may be an error between the prediction result and the gesture category pre-labeled by the image sample, and this error can be calculated using the cross-entropy loss function (Cross-Entropy Loss).

[0056] A similar approach can be used to calculate the errors of different subtask branches. Subsequently, by weighted summing the errors of each subtask branch, the overall model error can be obtained. This comprehensive error can be used to calibrate the overall model composed of the feature extraction network and each subtask branch, thereby completing the sample training process.

[0057] In one embodiment, to further improve gesture recognition accuracy, the training sample set for the gesture classification branch can be further subdivided into two refined sample sets: those with the palm facing the image capture device and those with the back of the hand facing the image capture device. Specifically, referring to FIG4 , a first sample set representing an open hand and a second sample set representing a closed hand can be obtained from the training sample set based on pre-labeled gesture category labels. For the first sample set, a first subset containing palm features and a second subset not containing palm features can be obtained. The inclusion of palm features can be understood as the palm being detectable from the image sample, indicating that the palm is facing the image capture device. The absence of palm features can be understood as the inability to detect the palm from the image sample, possibly due to the back of the hand facing the image capture device, or the palm being obscured by grasping another object while facing the image capture device. In this case, the palm feature is still classified as not containing palm features. Generally speaking, palm feature detection can be achieved through manual annotation.

[0058] After obtaining the first and second subsets, the image samples in the first subset can be labeled with a first label, the image samples in the second subset can be labeled with a second label, and the image samples in the second sample set can be labeled with a third label, wherein the first label, the second label, and the third label are mutually different. In this way, although all of them represent an open hand, by detecting the palm features, they can be further refined into two different labels, the first label and the second label. For a closed hand, a third label can be labeled that is different from both labels.

[0059] Subsequently, the model was trained using image samples labeled with three types of labels, and the resulting gesture recognition model can accurately identify situations where the hand is open and the palm is facing the image acquisition device. It can also accurately identify situations where the hand is open and the palm is facing the image acquisition device, but the palm is blocked by other objects held in the hand. This can improve the gesture recognition accuracy of the model and can be applied to complex application scenarios, thereby avoiding commands being triggered incorrectly. For example, under normal circumstances, the automatic photo-taking function will only be triggered when it is detected that the hand is open and the palm is facing the image acquisition device, and the palm features cannot be blocked. The technical solution provided by the present disclosure can accurately distinguish various situations and improve the accuracy of command execution.

[0060] It should be noted that although the gesture classification branch can initially identify whether the palm is facing the image acquisition device through refined training samples, if this recognition result conflicts with the palm orientation classification branch's recognition result, the palm orientation classification branch's recognition result can generally be used. Of course, in actual applications, a certain weight can also be used to make a trade-off, which can be flexibly determined based on the application scenario.

[0061] The technical solution provided by one or more embodiments of the present disclosure can analyze gesture features from different dimensions through multiple sub-task branches when identifying whether a gesture image contains a gesture representing a specified action. Specifically, the sub-task branch may include at least a gesture category classification branch and a palm orientation classification branch. Through different sub-task branches, different attribute information of gesture features can be obtained. By combining these attribute information, it can be more accurately determined whether a gesture image contains a gesture representing a specified action. In addition, by sharing gesture features through different sub-task branches, the model architecture can be simplified, and the use of multiple independent classification models can be avoided, thereby improving the efficiency of gesture recognition while ensuring the accuracy of gesture recognition.

[0062] In specific application scenarios, by segmenting the training samples, it is also possible to accurately identify whether the palm features are blocked, thereby ensuring the correct triggering of the command and reducing the false touch rate of the command.

[0063] Referring to FIG5 , an embodiment of the present disclosure further provides a gesture recognition device, the device comprising:

[0064] A feature extraction unit 100 is configured to acquire a gesture image and extract gesture features from the gesture image. The gesture features serve as shared features for multiple subtask branches for implementing gesture recognition. The multiple subtask branches include at least a gesture category classification branch and a palm orientation classification branch.

[0065] A first processing unit 200 is configured to process the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category;

[0066] A second processing unit 300 is configured to process the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation;

[0067] The gesture determination unit 400 is configured to determine whether the gesture image includes a gesture representing a designated action based on the first sub-attribute information and the second sub-attribute information.

[0068] The specific processing logic of each functional module can be found in the description of the aforementioned method implementation method, which will not be repeated here.

[0069] The various units described in the above embodiments can be implemented by computer chips or products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0070] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0071] Please refer to FIG6 . An embodiment of the present disclosure further provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program. When the computer program is executed by the processor, the above-mentioned gesture recognition method is implemented.

[0072] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program. When the computer program is executed by a processor, the above-mentioned gesture recognition method is implemented.

[0073] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0074] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing, thereby implementing the methods in the aforementioned method embodiments.

[0075] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0076] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). The storage medium can also include a combination of the above-mentioned types of memory.

[0077] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, equipment, and storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant portions, refer to the descriptions of the method embodiments.

[0078] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

[0079] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A gesture recognition method, comprising: Acquire a gesture image, and extract gesture features of the gesture image, wherein the gesture features are used as shared features of multiple subtask branches for implementing gesture recognition; The multiple subtask branches at least include a gesture category classification branch and a palm orientation classification branch; Processing the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category; Processing the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation; Based on the first sub-attribute information and the second sub-attribute information, it is determined whether the gesture image includes a gesture representing a designated action.

2. The method according to claim 1, wherein processing the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category comprises: Expanding the number of channels of the gesture features, and reshaping the gesture features after the number of channels has been expanded in their respective channels to generate channel reshaping features; Compressing the channel reshaping feature to obtain a one-dimensional feature with a length equal to the number of expanded channels; Based on the one-dimensional feature, first sub-attribute information representing the gesture category is generated.

3. The method according to claim 2, wherein generating first sub-attribute information characterizing the gesture category based on the one-dimensional feature comprises: Calculating the probability distribution vector of the one-dimensional feature in the gesture category classification task; A classification result of the gesture category represented by the probability distribution vector is determined, and first sub-attribute information representing the gesture category is generated based on the classification result.

4. The method according to claim 1, wherein the method further comprises: For a set of training samples suitable for the gesture category classification branch, from the training samples Obtaining a first sample set representing an open hand and a second sample set representing a closed hand from the set; In the first sample set, a first subset including palm features is obtained, and a second subset not including palm features is obtained; Annotate the image samples in the first subset with a first label, annotate the image samples in the second subset with a second label, and annotate the image samples in the second sample set with a third label; The first label, the second label and the third label are different from each other.

5. The method according to claim 1, wherein processing the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation comprises: Performing convolution processing on the gesture features with unchanged feature dimensions, and reshaping the gesture features after the convolution processing in their respective channels to generate channel reshaping features; Compressing the channel reshaping feature to obtain a one-dimensional feature with a length equal to the number of expanded channels; Based on the one-dimensional feature, second sub-attribute information representing the direction of the palm is generated.

6. The method according to claim 5, wherein generating second sub-attribute information characterizing the palm orientation based on the one-dimensional feature comprises: Calculating a probability distribution vector of the one-dimensional feature in a palm orientation classification task; A classification result of the palm orientation represented by the probability distribution vector is determined, and second sub-attribute information representing the palm orientation is generated based on the classification result.

7. The method according to claim 1, wherein the plurality of subtask branches further include a left-hand classification branch; the method further comprises: Processing the gesture feature through the left-right hand classification branch to generate third sub-attribute information representing the left-right hand category; Based on the first sub-attribute information, the second sub-attribute information, and the third sub-attribute information, it is determined whether the gesture image includes a gesture representing a designated action.

8. The method according to claim 7, wherein processing the gesture feature through the left-right hand classification branch to generate third sub-attribute information characterizing the left-right hand category comprises: Performing convolution processing on the gesture features with unchanged feature dimensions, and reshaping the gesture features after the convolution processing in their respective channels to generate channel reshaping features; Compressing the channel reshaping feature to obtain a one-dimensional feature with a length equal to the number of expanded channels; Based on the one-dimensional feature, third sub-attribute information characterizing the left and right hand categories is generated.

9. A gesture recognition device, comprising: A feature extraction unit, used to obtain a gesture image and extract gesture features of the gesture image, wherein the gesture features are used as shared features of multiple subtask branches for implementing gesture recognition; the multiple subtask branches at least include a gesture category classification branch and a palm orientation classification branch; A first processing unit, configured to process the gesture feature through the gesture category classification branch to generate first sub-attribute information representing the gesture category; A second processing unit, configured to process the gesture feature through the palm orientation classification branch to generate second sub-attribute information representing the palm orientation; A gesture determination unit is configured to determine whether the gesture image contains a gesture representing a designated action based on the first sub-attribute information and the second sub-attribute information.

10. An electronic device, wherein the electronic device comprises a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images