Gesture recognition method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a gesture recognition method and device, electronic equipment and a computer readable storage medium. The gesture recognition method comprises: acquiring an original image, wherein the original image contains a gesture; performing hand target detection on the original image to obtain a hand key feature data set; using the hand key feature data set as a training data set to train a preset initial gesture recognition model to obtain a basic gesture recognition model; performing pruning processing on the basic gesture recognition model to obtain a standard gesture recognition model; acquiring a to-be-recognized image, and using the standard hand recognition model to recognize a gesture type in the to-be-recognized image. In the gesture recognition method provided by the application, a gesture recognition model is combined, the characteristics of each gesture type are deeply mined, the trained gesture recognition model is pruned and optimized according to task requirements, and the accuracy of the model is improved without additional increase in computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, and particularly relates to a gesture recognition method and device, electronic equipment and a computer readable storage medium. BACKGROUND

[0002] Gesture recognition is one of the current gradually popular human-computer interaction methods, which can be applied to various fields such as smart home, intelligent transportation, virtual reality, smart TV, online education, air keyboard, motion game, intelligent cockpit, intelligent fitness mirror, intelligent logistics, etc.

[0003] Due to the diversity of gestures, a large number of types need to be recognized, and in a real-time dynamic scene, motion blur will also occur during the switching process of gestures, so as to cause the gesture recognition algorithm to have low accuracy and poor robustness in the application of technology landing. In order to improve the recognition accuracy, some researchers have proposed some complex network structures such as SPP-Net, R-CNN, FasterRCNN, etc., but due to the large amount of model calculation, these are not suitable for deployment to mobile terminals, and some researchers use lightweight networks, but the accuracy cannot achieve the effect of algorithm landing. SUMMARY

[0004] Therefore, the present application provides a gesture recognition method, device, electronic equipment and computer readable storage medium, which can improve the accuracy of gesture recognition without additional computing resources.

[0005] In a first aspect, an embodiment of the present application provides a gesture recognition method, comprising:

[0006] obtaining an original image, wherein the original image contains a gesture;

[0007] performing hand target detection on the original image to obtain a hand key feature dataset;

[0008] training a preset initial gesture recognition model using the hand key feature dataset as a training dataset to obtain a basic gesture recognition model;

[0009] performing pruning processing on the basic gesture recognition model to obtain a standard gesture recognition model;

[0010] obtaining a to-be-recognized image and recognizing a gesture type in the to-be-recognized image using the standard gesture recognition model.

[0011] In an embodiment, the training of the preset initial gesture recognition model using the hand key feature dataset as a training dataset to obtain a basic gesture recognition model comprises:

[0012] adopt a lightweight MobileNetV2 network model as a main network of the preset initial gesture recognition model;

[0013] intermediate connection of a regression branch containing preset gesture key features in the main network, and connection of a classification branch containing multiple preset gestures at the end of the main network;

[0014] freeze the main network, and train weights of the regression branch and the classification branch separately;

[0015] after the preset initial gesture recognition model converges, open the main network, adjust weights of the preset initial gesture recognition model, and obtain the basic gesture recognition model.

[0016] In an embodiment, the multiple preset gestures at least include a heart gesture, an OK gesture, a palm opening gesture, a fist gesture, a salute gesture, a thumbs-up gesture, a thumbs-down gesture, a first rock gesture, an index finger pointing gesture, a two-finger gesture, a victory gesture, a large V gesture, a phone call gesture, a cabbage gesture, a four-finger gesture, a first gun gesture, a second rock gesture, a palm-down gesture, a swearing gesture, a second gun gesture, a Spiderman gesture, and a flirting gesture.

[0017] In an embodiment, the pruning processing of the basic gesture recognition model to obtain a standard gesture recognition model includes deleting the regression branch in the basic gesture recognition model to obtain the standard gesture recognition model.

[0018] In an embodiment, the hand target detection on the original image to obtain a hand key point dataset includes:

[0019] hand target detection on the original image to determine hand position information;

[0020] determination of a hand target frame according to the hand position information;

[0021] cutting of a hand region of interest in the hand target frame;

[0022] uniform size processing of the hand region of interest to obtain the hand key feature dataset.

[0023] In an embodiment, the hand position information at least includes a hand center point coordinate, a hand height, and a hand width.

[0024] In an embodiment, the determination of the hand target frame according to the hand position information includes comparison of the hand height and the hand width, determination of a long side of the hand height and the hand width as a side length of the hand target frame, and obtaining of the hand target frame.

[0025] Secondly, one embodiment of this application provides a gesture recognition device, comprising: an acquisition module for acquiring an original image, wherein the original image contains a gesture; a detection module for performing hand target detection on the original image to obtain a hand key feature dataset; a training module for training a preset initial gesture recognition model using the hand key feature dataset as a training dataset to obtain a basic gesture recognition model; a processing module for pruning the basic gesture recognition model to obtain a standard gesture recognition model; and a recognition module for acquiring an image to be recognized and using the standard hand recognition model to recognize the gesture type in the image to be recognized.

[0026] Thirdly, one embodiment of this application provides an electronic device, the electronic device including a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, causing the electronic device to perform the steps of the gesture recognition method as described in the first aspect.

[0027] Fourthly, one embodiment of this application provides a computer-readable storage medium that stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the steps of the gesture recognition method as described in the first aspect.

[0028] The gesture recognition method, apparatus, electronic device, and computer-readable storage medium provided in this application use the MobileNetV2 lightweight network as the backbone network of the model. The gesture recognition task is decomposed into a dual-task training of 23 gesture classifications and 21 gesture key points. By fusing the 21 gesture key point features on the basis of the 23 gesture classification features, the model can learn richer semantic features. At the same time, the accuracy of the model is improved without increasing the computational overhead. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the steps of a gesture recognition method according to an embodiment of this application.

[0030] Figure 2 for Figure 1 The diagram shows a breakdown of the first step of the gesture recognition method.

[0031] Figure 3 for Figure 1 The diagram shows a breakdown of the first step of the gesture recognition method.

[0032] Figure 4 This is a schematic diagram of the functional modules of a gesture recognition device according to an embodiment of this application.

[0033] Figure 5An architecture schematic diagram of an electronic device according to an embodiment of the present application.

[0034] Main element symbol description

[0035] Gesture recognition apparatus 10

[0036] Acquisition module 11

[0037] Detection module 12

[0038] Training module 13

[0039] Processing module 14

[0040] Recognition module 15

[0041] Electronic device 20

[0042] Processor 21

[0043] Memory 22

[0044] The following detailed description will further describe the present application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application.

[0046] It should be noted that “at least one” in the embodiments of the present application means one or more, and more means two or more. Unless otherwise defined, all technical and scientific terms used in the present application have the same meanings as those commonly understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the present application.

[0047] It should be noted that in the embodiments of the present application, the terms “first”, “second”, and the like are only used for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying sequence. The features limited by “first”, “second” can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the terms “exemplary” or “for example” are used to mean as an example, illustration, or description. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the terms “exemplary” or “for example” are used in the specific manner to present the relevant concept.

[0048] Please refer to Figure 1A flowchart of a gesture recognition method provided by an embodiment of the present application is shown.

[0049] Specifically, the gesture recognition method can include:

[0050] S100, obtaining an original image.

[0051] In the embodiment, the original image containing a hand gesture can be directly obtained by using an image acquisition device, for example, the image acquisition device is a camera, a mobile terminal with a camera, etc. The original image can also be obtained from a storage device, for example, the storage device is a U disk, etc. The original image can also be obtained from a website server through a network. The embodiment does not limit the manner of obtaining the original image.

[0052] S200, performing hand target detection on the original image to obtain a hand key feature data set.

[0053] Further combining Figure 2 As shown in the figure, step S200 can include:

[0054] S210, performing hand target detection on the original image to determine hand position information.

[0055] In the embodiment, the hand position information at least includes a hand center point coordinate, a hand height, and a hand width.

[0056] S220, determining a hand target frame according to the hand position information.

[0057] In the embodiment, in order to solve the problem that the sizes of the detected hand target frames are different and the aspect ratios are different, the hand height and the hand width are compared, the long side of the hand height and the hand width is determined as the side length of the hand target frame, and the hand target frame is obtained.

[0058] S230, intercepting a hand interest region in the hand target frame.

[0059] S240, performing uniform size processing on the hand interest region to obtain a hand key feature data set.

[0060] In the embodiment, the hand interest region is unified to a size of 96*96, so that the hand interest region in each original image in the hand key feature data set is uniform in size, thereby improving the accuracy and speed of the model training.

[0061] S300, training a preset initial gesture recognition model by using the hand key feature data set as a training data set to obtain a basic gesture recognition model.

[0062] Further combining Figure 3 As shown in the figure, step S300 can include:

[0063] S310, adopting a lightweight MobileNetV2 network model as a preset initial gesture recognition model backbone network.

[0064] S320, in the middle of the backbone network, a regression branch containing preset gesture key features is connected, and at the end of the backbone network, a classification branch containing multiple preset gestures is connected.

[0065] In this embodiment, the gesture recognition task is decomposed into a 23-gesture classification and a 21-gesture key point regression dual-task training by using a lightweight MobileNetV2 network model. The purpose is to fuse the 21-gesture key point features on the basis of the 23-gesture classification features, so that the model can learn more rich semantic features, and the model accuracy can be improved without additional computational overhead. Among them, the regression branch and the classification branch both use a 1*1 convolution layer to replace the conventional full connection layer, so as to reduce the parameter amount and the calculation overhead. Experiments show that after adding the regression branch in the backbone network, the model recognition accuracy of the following preset 23 gestures is increased by an average of 17.4%.

[0066] Specifically, the multiple preset gestures at least include: a heart gesture, an OK gesture, a palm opening gesture, a fist gesture, a salute gesture, a thumbs-up gesture, a thumbs-down gesture, a first rock gesture, an index finger pointing gesture, a two-finger gesture, a victory gesture, a big V gesture, a phone call gesture, a cabbage gesture, a four-finger gesture, a first gun gesture, a second rock gesture, a palm-down gesture, a swear gesture, a second gun gesture, a spiderman gesture, and a flirt gesture.

[0067] S330, freezing the backbone network, and training the weights of the regression branch and the classification branch separately.

[0068] In this embodiment, the preset initial gesture recognition model converges quickly and tends to be stable after 10 epochs.

[0069] S340, after the preset initial gesture recognition model converges, the backbone network is opened, the weights of the preset initial gesture recognition model are adjusted, and a basic gesture recognition model is obtained.

[0070] In this embodiment, fine-tuning the weights of the preset initial gesture recognition model can make the weights of the basic gesture recognition model fit the training data set to a higher degree, so as to increase the accuracy of the basic gesture recognition model.

[0071] S400, pruning the basic gesture recognition model to obtain a standard gesture recognition model.

[0072] In the embodiment, according to the training task requirement, the regression branch in the basic gesture recognition model is deleted to obtain the standard gesture recognition model. In the model inference stage, only the backbone network and the classification branch are retained, the model parameter quantity can be reduced and the model inference speed can be improved without affecting the model recognition accuracy. Experiments show that after pruning, the total parameter quantity is reduced by 48682, and the forward inference time consumption of the model on the honor v30 model is reduced by 0.282ms.

[0073] S500, acquiring an image to be recognized, and recognizing a gesture type in the image to be recognized by using the standard gesture recognition model.

[0074] The gesture recognition method in the above embodiment uses the MobileNetV2 lightweight network as the backbone network of the model, decomposes the gesture recognition task into a 23-gesture classification and a 21-key point regression dual task training, and fuses the 21 gesture key point features on the basis of the 23 gesture classification features, so that the model can learn more rich semantic features, and the accuracy of the model can be improved without additional calculation overhead.

[0075] Further combining Figure 4 As shown in FIG. 10, a gesture recognition device 10 is provided in an embodiment of the present application.

[0076] Specifically, the gesture recognition device 10 can include: an acquisition module 11 configured to acquire an original image, wherein the original image contains a gesture; a detection module 12 configured to perform hand target detection on the original image to obtain a hand key feature dataset; a training module 13 configured to train a preset initial gesture recognition model by using the hand key feature dataset as a training dataset to obtain a basic gesture recognition model; a processing module 14 configured to perform pruning processing on the basic gesture recognition model to obtain a standard gesture recognition model; and an identification module 15 configured to acquire an image to be recognized, and recognize a gesture type in the image to be recognized by using the standard gesture recognition model.

[0077] Please refer to Figure 5 , Figure 5 An architecture schematic diagram of an electronic device 20 is provided in an embodiment of the present application.

[0078] Specifically, the electronic device 20 includes a processor 21 and a memory 22, the memory 22 is configured to store computer instructions, and the processor 21 is configured to invoke the computer instructions in the memory 22, so that the electronic device 20 performs related steps of the gesture recognition method of the above embodiment.

[0079] For example, the computer instructions can be divided into one or more modules / units, one or more modules / units are stored in the memory 22 and executed by the processor 21. One or more modules / units can be a series of computer instruction segments capable of completing a specific function, and the instruction segments are used to describe the execution process of the computer instructions in the electronic device 20. For example, the computer instructions can be divided into Figure 4 The acquisition module 11, the detection module 12, the training module 13, the processing module 14, and the identification module 15 shown.

[0080] The electronic device 20 can be a desktop computer, a notebook, a palm computer, an industrial computer, a tablet computer, a server, and the like. Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 20, and does not constitute a limitation on the electronic device 20, and can include more or less components than the diagram, or combine certain components, or different components, for example, the electronic device 20 can also include an input / output device, a network access device, a bus, and the like.

[0081] The processor 21 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor, a single-chip computer, and the processor 21 can also be any conventional processor, and the like.

[0082] The memory 22 can be used to store computer instructions and / or modules / units, and the processor 21 implements various functions of the electronic device 20 by running or executing the computer instructions and / or modules / units stored in the memory 22 and calling data stored in the memory 22. The memory 22 can mainly include a storage program area and a storage data area, wherein the storage program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the storage data area can store data (such as audio data) created according to the use of the electronic device 20, etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other nonvolatile solid-state storage device.

[0083] An embodiment of the present application further provides a computer readable storage medium, which stores computer instructions, and when the computer instructions are run on the electronic device 20, the electronic device 20 performs the related steps of the gesture recognition method of the above-mentioned embodiment. The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and the like, which are various media capable of storing program codes.

[0084] The gesture recognition method, device, electronic device 20 and computer readable storage medium provided by the embodiments of the present application use the MobileNetV2 lightweight network as the backbone network of the model, decompose the gesture recognition task into a 23-class gesture classification and a 21-key point regression dual-task training, and fuse the 21 gesture key point features on the basis of the 23-class gesture classification features, so that the model can learn more rich semantic features, and meanwhile, the accuracy of the model is improved without additional calculation overhead.

[0085] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present application, and are not used as a limitation on the present application, and as long as the above embodiments are appropriately changed and varied within the scope of the spirit of the present application, they fall within the scope of the present application.

Claims

1. A gesture recognition method, characterized by, The method comprises the following steps: acquiring an original image, wherein the original image contains a hand gesture; performing hand target detection on the original image to obtain a hand key feature data set; training a preset initial gesture recognition model using the hand key feature data set as a training data set to obtain a basic gesture recognition model, including: using a lightweight MobileNetV2 network model as a backbone network of the preset initial gesture recognition model; connecting a regression branch containing preset hand gesture key features in the middle of the backbone network, and connecting a classification branch containing a plurality of preset hand gestures at the end of the backbone network; freezing the backbone network, and training the weights of the regression branch and the classification branch separately; after the preset initial gesture recognition model converges, opening the backbone network, adjusting the weights of the preset initial gesture recognition model to obtain the basic gesture recognition model; performing pruning processing on the basic gesture recognition model to obtain a standard gesture recognition model, including: deleting the regression branch in the basic gesture recognition model to obtain the standard gesture recognition model; acquiring a to-be-recognized image, and recognizing the hand gesture type in the to-be-recognized image using the standard gesture recognition model.

2. The gesture recognition method of claim 1, wherein, The plurality of preset hand gestures at least include: a heart gesture, an OK gesture, a palm opening gesture, a fist gesture, a salute gesture, a thumbs-up gesture, a thumbs-down gesture, a first rock gesture, an index finger pointing gesture, a two-finger stretching gesture, a victory gesture, a big V gesture, a phone call gesture, a cabbage gesture, a four-finger gesture, a first gun gesture, a second rock gesture, a palm-down gesture, a swearing gesture, a second gun gesture, a Spiderman gesture, and a flirting gesture.

3. The gesture recognition method of claim 1, wherein, The hand target detection on the original image to obtain a hand key point data set comprises the following steps: performing hand target detection on the original image to determine hand position information; determining a hand target box according to the hand position information; cutting a hand region of interest in the hand target box; performing uniform size processing on the hand region of interest to obtain the hand key feature data set.

4. The gesture recognition method of claim 3, wherein, The hand position information at least includes: a hand center point coordinate, a hand height, and a hand width.

5. The gesture recognition method of claim 4, wherein, The determination of the hand target box according to the hand position information comprises: comparing the hand height with the hand width, determining the longer side of the hand height and the hand width as the side length of the hand target box to obtain the hand target box.

6. A gesture recognition apparatus, characterized by The method comprises the following steps: an acquisition module, configured to acquire an original image, wherein the original image contains a hand gesture; a detection module, configured to perform hand target detection on the original image to obtain a hand key feature data set; The training module is configured to train a preset initial gesture recognition model by using the hand key feature dataset as a training dataset to obtain a basic gesture recognition model, including: adopting a lightweight MobileNetV2 network model as a backbone network of the preset initial gesture recognition model; connecting a regression branch containing preset hand gesture key features in the middle of the backbone network, and connecting a classification branch containing multiple preset gestures at the end of the backbone network; freezing the backbone network, and training weights of the regression branch and the classification branch separately; after the preset initial gesture recognition model converges, opening the backbone network, adjusting weights of the preset initial gesture recognition model to obtain the basic gesture recognition model; The processing module is configured to perform pruning processing on the basic gesture recognition model to obtain a standard gesture recognition model, including: deleting the regression branch in the basic gesture recognition model to obtain the standard gesture recognition model; The recognition module is configured to obtain a to-be-recognized image, and recognize a gesture type in the to-be-recognized image by using the standard gesture recognition model.

7. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory is configured to store instructions, and the processor is configured to call the instructions in the memory, so that the electronic device performs the steps of the gesture recognition method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and when the computer instructions run on the electronic device, the electronic device performs the steps of the gesture recognition method in any one of claims 1 to 5.

Images