A gesture recognition method based on a head-mounted display device

Abstract

The application provides a gesture recognition method based on a head-mounted display device, comprising: inputting an image into a hand key point detection model, the hand key point detection model comprising a plurality of stacked feature extraction modules, sequentially adopting the plurality of stacked feature extraction modules to perform feature extraction on the input image to obtain a feature map of a preset size; expanding the feature map of the preset size into a one-dimensional vector through a dimension expansion layer, and adopting a fully connected layer to perform regression prediction of hand key points to obtain coordinates of the hand key points; and obtaining a gesture classification result in the image through a combination relationship between the coordinates of the hand key points. The application combines a hand detection model and a hand key point detection model to realize detection of a gesture, since the network structures of the hand detection model and the hand key point detection model are substantially the same, and only differ in a task head, the full network structure of the gesture detection of the application has small calculation amount and low power consumption under the condition of meeting detection accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of head-mounted display devices, and particularly relates to a gesture recognition method based on a head-mounted display device. BACKGROUND

[0002] Gesture recognition is a technology that recognizes user intentions and instructions by analyzing and understanding human body movements and gestures. It can capture and analyze human posture, gesture movements and spatial position information through the use of sensors, computer vision and machine learning technologies, thereby enabling natural interaction with computer systems.

[0003] Gesture recognition technology based on head-mounted display devices is an application that combines augmented reality (AR) and gesture recognition. Head-mounted display devices are smart glasses devices that can superimpose virtual information on the real world, allowing users to interact with virtual and augmented reality content.

[0004] Applying gesture recognition technology to head-mounted display devices can provide users with a more intuitive, natural and immersive interaction experience. By recognizing user gestures, head-mounted display devices can understand user intentions and instructions and accordingly present virtual content, perform operations or provide relevant information. This interaction method can free users from the limitations of traditional input devices and make it more convenient for them to interact with virtual and real worlds.

[0005] Existing head-mounted display device-based gesture recognition methods based on pure vision generally directly input video streams, then frame the images for recognition, detect the hands in the picture, and then perform gesture recognition. This requires real-time running of hand detection and tracking algorithms, which requires a large amount of computing power. However, head-mounted display devices have limited computing power and inference performance, which greatly affects user and head-mounted display device gesture interaction.

[0006] Existing technologies also have some lightweight gesture recognition models, but meeting the lightweight requirements also reduces the model parameter quantity and calculation amount, which reduces the accuracy of gesture recognition and also affects the user's interaction experience. Therefore, existing gesture recognition methods either have high power consumption and high accuracy, or have low power consumption accompanied by a certain decrease in accuracy. SUMMARY

[0007] In order to overcome the above technical defects, the purpose of the present application is to provide a gesture recognition method based on a head-mounted display device that is more lightweight and meets the precision requirements of use.

[0008] The application discloses a gesture recognition method based on a head-mounted display device, comprising the following steps: inputting an image into a hand key point detection model, wherein the hand key point detection model comprises a plurality of stacked feature extraction modules; sequentially performing feature extraction on the input image by using the plurality of stacked feature extraction modules to obtain a feature map with a preset size; expanding the feature map with the preset size into a one-dimensional vector through a dimension expansion layer, and performing regression prediction of hand key points by using a fully connected layer to obtain coordinates of the hand key points; and obtaining a gesture classification result in the image through a combination relationship between the coordinates of the hand key points.

[0009] Preferably, the step of sequentially performing feature extraction on the input image by using the plurality of stacked feature extraction modules comprises the following steps: for a first feature extraction module in the plurality of stacked feature extraction modules: performing dimension lifting on the image by using a channel expansion layer to obtain a feature map after dimension lifting; performing group convolution processing on the feature map after dimension lifting to obtain a feature map after group convolution; adding the feature map after group convolution and the feature map after dimension lifting, and then performing activation through an activation layer to obtain a first feature map of the first feature extraction module.

[0010] Preferably, the step of sequentially performing feature extraction on the input image by using the plurality of stacked feature extraction modules comprises the following steps: for other feature extraction modules in the plurality of stacked feature extraction modules: performing down-sampling on a feature map output by a previous feature extraction module to obtain a feature map after down-sampling; performing group convolution processing on the feature map after down-sampling to obtain a feature map after group convolution; adding the feature map after down-sampling and the feature map after group convolution, and then performing activation through an activation layer to obtain a feature map of the other feature extraction module.

[0011] Preferably, the group convolution processing comprises the following steps: inputting the feature map into a group convolution module to perform feature extraction by using a multi-size kernel; inputting an output feature of the feature extraction by using the multi-size kernel into a normalization layer and an activation function layer to output a feature map after group convolution.

[0012] Preferably, the step of inputting the output feature of the feature extraction by using the multi-size kernel into the normalization layer and the activation function layer to output the feature map after group convolution comprises the following steps: for a last feature extraction module in the plurality of stacked feature extraction modules: inputting the output feature of the feature extraction by using the multi-size kernel into the normalization layer to obtain the feature map after group convolution; adding the feature map after down-sampling and the feature map after group convolution, and then performing activation through an activation layer to obtain a feature map of the last feature extraction module.

[0013] Preferably, the feature map is input into the group convolution module for multi-size kernel feature extraction, which comprises: channel disassembling the feature map to obtain multiple groups of channels, and the convolution kernel of each group of channels is different; performing group convolution in each group of channels respectively, wherein the number of groups is equal to the number of channels; then splicing the channels of the result of each group convolution; finally, transforming the feature map through an attention mechanism network.

[0014] Preferably, after the feature map is input into the group convolution module for multi-size kernel feature extraction, it further comprises: channel scrambling to mix the features extracted from different size kernels.

[0015] Preferably, before the image is input into the hand key point detection model, it further comprises: detecting the hand region image in the image through a hand detection model; tracking the hand region image through a tracker, and inputting the tracked hand region image into the hand key point detection model.

[0016] Preferably, the hand detection model comprises: changing or deleting the dimension expansion layer and / or the full connection layer in the hand key point detection model to obtain the hand detection model.

[0017] Preferably, the hand region image in the new image is detected through the hand detection model every N frames.

[0018] After the above technical scheme is adopted, compared with the prior art, the following beneficial effects are obtained:

[0019] 1. The gesture detection of the present application is realized by combining the hand detection model and the hand key point detection model. Since the network structures of the hand detection model and the hand key point detection model are roughly the same, only the task head is different, the whole network structure of the gesture detection of the present application has small calculation amount and low power consumption under the condition of meeting the detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The flowchart of the gesture recognition method based on the head-mounted display device provided by the present application

[0021] Figure 2 The network structure diagram of the hand key point detection model provided by the present application

[0022] Figure 3 The flowchart of the hand detection process before the hand key point detection provided by the present application DETAILED DESCRIPTION

[0023] The advantages of the present application are further described below in combination with the drawings and specific embodiments.

[0024] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements throughout the description. The following exemplary embodiments are not representative of all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0025] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the description of the present disclosure and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0026] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a particular order or hierarchy. These terms are used merely for the purpose of distinguishing between two or more information. For example, without departing from the scope of the present disclosure, a first information can be termed a second information, and similarly, a second information can be termed a first information. The word "if" as used herein means "when" or "upon" or "in response to the determination" depending on the context.

[0027] In the description of the present disclosure, it should be understood that the terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present disclosure and simplify the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present disclosure.

[0028] In the description of the present disclosure, unless otherwise specified and limited, it should be explained that the terms "mounting", "connecting", "connection" should be understood broadly, for example, it can be mechanical connection or electrical connection, it can be the communication between the internal elements, it can be direct connection or indirect connection through intermediate medium, and the specific meaning of the above terms can be understood by the person skilled in the art according to the specific circumstances.

[0029] In the subsequent description, the suffix such as "module", "component", or "unit" used to represent elements is only for the convenience of the description of the present disclosure, and has no specific meaning by itself. Therefore, "module" and "component" can be used interchangeably.

[0030] Referring to the accompanying drawings Figures 1-2 The application discloses a gesture recognition method based on a head-mounted display device, which is used for recognizing a user's gesture in real time through the head-mounted display device after the user wears the head-mounted display device, so that the use of the head-mounted display device is more convenient and humanized. The gesture recognition method comprises the following steps:

[0031] S100, inputting an image into a hand key point detection model, wherein the hand key point detection model comprises a plurality of stacked feature extraction modules, and the input image is sequentially subjected to feature extraction by the plurality of stacked feature extraction modules to obtain a feature map with a preset size;

[0032] S200, expanding the feature map with the preset size into a one-dimensional vector through a dimension expansion layer, and performing regression prediction of hand key points through a full connection layer to obtain coordinates of the hand key points;

[0033] S300, obtaining a gesture classification result in the image through a combination relationship between the coordinates of the hand key points.

[0034] The key points to be detected in steps S200-S300 are usually 21 hand key points of a hand, that is, after 21 hand key points are obtained, different gesture classification results can be obtained based on the combination of the key points (that is, the position relationship between the key points).

[0035] Referring to the accompanying drawings Figure 2 The left part of the network structure, in step S100, the input image is subjected to feature extraction by the hand key point detection model through the feature extraction modules, that is, the input image is sequentially subjected to feature extraction by a plurality of stacked feature extraction modules, and after the feature extraction, a feature map with a preset size is obtained.

[0036] Specifically, for the first feature extraction module in the plurality of stacked feature extraction modules, the feature extraction step comprises the following steps.

[0037] S111, performing dimension upgrading on the image through a channel expansion layer to obtain an upgraded feature map;

[0038] S112, performing group convolution processing on the upgraded feature map to obtain a group-convoluted feature map;

[0039] S113, adding the group-convoluted feature map and the upgraded feature map, and then performing activation through an activation layer to obtain a first feature map of the first feature extraction module.

[0040] Specifically, for the first feature extraction module, preferably, the input image is first processed by a channel expansion layer for dimensionality increasing, then group convolution is performed, and then the feature map after dimensionality increasing and the feature map after group convolution are added and activated, so as to output the first feature map of the first feature extraction module. Here, the addition refers to adding corresponding data of the two feature maps through a short connection path. If the dimensions of the two feature maps are different, a linear mapping needs to be performed on one of the added values to match the dimensions. This operation is to ensure the feature extraction capability and acceleration convergence capability of the feature extraction module.

[0041] For other feature extraction modules in the plurality of stacked feature extraction modules, the feature extraction step is:

[0042] S121, down-sampling the feature map output by the previous feature extraction module to obtain a down-sampled feature map;

[0043] S122, performing group convolution on the down-sampled feature map to obtain a group-convoluted feature map;

[0044] S123, adding the down-sampled feature map and the group-convoluted feature map, and then activating through an activation layer to obtain the feature map of the other feature extraction module.

[0045] Specifically, for the first feature extraction module in the multi-module stack, the input image is first processed by a channel expansion layer (such as expand_channels) for dimensionality increasing. For other feature extraction modules in the multi-module stack, the feature map output by the previous feature extraction module is down-sampled (such as down_sampling). That is, channel expansion is only used in the first feature extraction module, and in other feature extraction modules, the feature map output by the previous feature extraction module is directly down-sampled. Then, the down-sampled feature map is group-convoluted, and then the down-sampled feature map and the group-convoluted feature map are added and activated (such as Activate), so as to output the feature map of the other feature extraction module. In step S121, the down-sampling process preferably adopts the ways of maximum pooling, convolution with a step or average pooling, which is not limited here.

[0046] In the drawings: Figure 2In step S112 and step S122, the group convolution processing includes inputting the feature map into the group convolution module for multi-size kernel feature extraction; inputting the output feature of the multi-size kernel feature extraction into a normalization layer (such as BN) and an activation function layer (such as Activate) to output the feature map after group convolution. Specifically, the up-sampled feature map in the first feature extraction module or the down-sampled feature map of the other feature extraction modules is input into the group convolution module for multi-size kernel feature extraction, and then input into the normalization layer and the activation function layer, and finally the feature map after group convolution is output. In some embodiments, before the up-sampled feature map in the first feature extraction module or the down-sampled feature map of the other feature extraction modules is input into the group convolution module, the up-sampling is performed through a 1X1 convolution kernel, and then the up-sampled feature map is input into the group convolution module for multi-size kernel feature extraction. In some embodiments, the feature map after multi-size kernel feature extraction is first shuffled (such as Shuffle_Channels) to mix the features extracted from different size kernels, and then input into the normalization layer and the activation function layer, and finally the feature map after group convolution is output.

[0047] In some embodiments, the group convolution module corresponding to the last feature extraction module in the plurality of stacked feature extraction modules includes: inputting the feature map into the group convolution module for multi-size kernel feature extraction.

[0048] The output feature of the multi-size kernel feature extraction is input into the normalization layer to obtain the feature map after group convolution. Therefore, for the last feature extraction module in the plurality of stacked feature extraction modules, the feature extraction step is:

[0049] The down-sampled feature map and the feature map after group convolution are added, and then the activation layer is activated to obtain the feature map of the last feature extraction module.

[0050] In the group convolution module corresponding to the last feature extraction module, the activation function layer is not included, and only the output feature of the multi-size kernel feature extraction is input into the normalization layer to obtain the feature map after group convolution. Then, the feature map output by the previous feature extraction module is down-sampled, and the down-sampled feature map is added to the feature map after group convolution corresponding to the last feature extraction module, and then input into the activation function layer to obtain the feature map of the last feature extraction module.

[0051] It should be noted that the multi-size kernel feature extraction in steps S112 and S122 includes the processing procedure of n group convolution modules, and the value of n can be freely adjusted. Wherein, referring to the right part of the network structure of the accompanying Figure 2 The feature extraction procedure of the multi-size kernel of each group convolution module includes:

[0052] S1021, channel splitting is performed on the feature map to obtain multiple groups of channels, and the convolution kernel of each group of channels is different;

[0053] S1022, group convolution is respectively performed in each group of channels, and the number of groups is equal to the number of channels;

[0054] S1023, then the results of each group convolution are spliced into channels;

[0055] S1024, finally the feature map is transformed through the attention mechanism network.

[0056] Wherein, for each input feature map, channel splitting (such as split_channels) is performed to split into multiple groups, at most 4 groups, the minimum size of the convolution kernel is 3x3, and the maximum size is 9x9, for example, 3x3, 5x5, 7x7 in the preferred embodiment, group convolution is respectively performed in each group of channels, then the results of each group convolution are spliced into channels (such as channels_cat), and finally the feature map after channel splicing is transformed through the attention mechanism network (such as SEnet). Because the group convolution with the number of groups equal to the number of channels is used, the information exchange between different features will be poor, so after the feature extraction of the multi-size kernel, channel shuffling is performed to mix the features extracted from different size kernels, that is, in the group convolution module, the multi-size kernel

[0057] The channel shuffling is performed before the feature input normalization layer and the activation function layer after the feature extraction, to mix the features extracted from different size kernels.

[0058] The above process is mainly aimed at the detection of hand key points after the hand image has been obtained. The detection of hand key points is based on the above-mentioned very light network structure, and can easily achieve real-time hand key point detection on a head-mounted display device (such as AR glasses). In actual interaction, the hand occupies a very small proportion in the screen of the head-mounted display device (such as AR glasses), that is, a large part of the background image is useless, and if the entire screen is calculated, the calculation amount will be increased. Therefore, the gesture recognition method of the present application further comprises the following steps before the key point detection: detecting the hand region image in the image by the hand detection model; tracking the hand region image by the tracker, and inputting the tracked hand region image into the hand key point detection model. Generally, when interacting with AR glasses, the hand detection model is called first to obtain the preselected region of the hand. Since the algorithm of the hand detection model is relatively time-consuming compared to the tracking algorithm, it will affect the power consumption of the overall gesture recognition method. Therefore, the present application adopts a detection combined with tracking manner. Since the tracking algorithm does not need to read the entire image, it only needs to track in the local region, and then input the tracked hand region image into the hand key point detection model for subsequent hand key point detection.

[0059] In some embodiments, the network structure of the hand detection of the present application adopts the same network structure as the network structure of the hand key point detection model described above, and only the task head is changed. The dimension expansion layer and / or the fully connected layer in the hand key point detection model are changed or deleted to obtain the hand detection model. That is, the dimension expansion layer and / or the fully connected layer can be directly deleted, or the dimension expansion layer and / or the fully connected layer can be changed. In this way, the calculation amount of the overall gesture detection network structure can be reduced.

[0060] Specifically, referring to FIG. 2, the hand detection model is used to detect the hand region in the image, and the hand region image is obtained. Then, the hand region image is input into the tracker to track the hand region, and the tracked hand region image is input into the hand key point detection model to detect the hand key points in the hand region image. Figure 3 When the gesture recognition starts, the hand detection model is used to detect the hand in the collected hand image frame, and then the bounding box of the hand region is obtained. Then, on the one hand, the tracker is initialized to track the hand, and on the other hand, the hand region image detected by the hand detection model is input into the hand key point detection model for detection. Next, the hand region in the subsequent hand image frame is extracted (i.e., the hand region image is obtained) by the tracking algorithm, and the tracked hand region image is input into the hand key point detection model for detection.

[0061] In the tracking process, the hand detection model is also used to detect the hand region image in a new image every N frames. Specifically, in the tracking process, if the target disappears, the hand detection model is called again for detection, and if the target does not disappear, the hand detection model is used to detect the hand region image in a new image every N frames, and then the hand tracking algorithm is used for hand tracking to update the gesture in real time, and the tracked hand region image is given to the hand key point detection model for detection. Preferably, the tracking algorithm used here is the MOSSE tracking algorithm.

[0062] In addition, in order to achieve better results, the hand detection model of the present application is trained using a transfer learning method. Specifically, a classification model is first created which is the same as the lightweight network of the feature extraction part of the hand key point detection model, and the classification model is trained on a dataset containing hands to obtain a classification model with good performance for hand classification. Then, the weights of this model are used to initialize the feature extraction part of the hand detection model and the hand key point detection model. Based on this method, compared with training from scratch, the model convergence speed and the final detection effect can be significantly improved.

[0063] The gesture recognition method of the present application is a lightweight neural network, which reduces the inference time of the model and ensures sufficient recognition accuracy without affecting the user experience. By optimizing the detection, tracking and recognition logic of the gesture recognition, the running power consumption is further reduced, and the real-time performance and reliability of the gesture recognition are significantly improved.

[0064] It should be noted that the embodiments of the present application have better implementation, and do not limit the present application in any form, and any skilled person in the art can change or modify the above disclosed technical content into equivalent effective embodiments, as long as it does not deviate from the content of the technical solution of the present application, and any modification or equivalent change and modification of the above embodiments according to the technical essence of the present application, all still belong to the scope of the technical solution of the present application.

Claims

1. A method of gesture recognition based on a head-mounted display device, the method comprising: The method comprises the following steps: inputting an image into a hand key point detection model, the hand key point detection model comprising a plurality of stacked feature extraction modules, sequentially performing feature extraction on the input image by using the plurality of stacked feature extraction modules to obtain a feature map of a preset size; expanding the feature map of the preset size into a one-dimensional vector through a dimension expansion layer, and performing regression prediction of hand key points by using a fully connected layer to obtain coordinates of the hand key points; obtaining a hand gesture classification result in the image through a combination relationship between the coordinates of the hand key points; wherein, sequentially performing feature extraction on the input image by using the plurality of stacked feature extraction modules comprises: for a first feature extraction module in the plurality of stacked feature extraction modules: dimensionally upgrading the image through a channel expansion layer to obtain an upgraded feature map; performing group convolution processing on the upgraded feature map to obtain a group-convoluted feature map; adding the group-convoluted feature map and the upgraded feature map, and then performing activation through an activation layer to obtain a first feature map of the first feature extraction module; the sequentially performing feature extraction on the input image by using the plurality of stacked feature extraction modules comprises: for other feature extraction modules in the plurality of stacked feature extraction modules: performing down-sampling on a feature map output by a previous feature extraction module to obtain a down-sampled feature map; performing group convolution processing on the down-sampled feature map to obtain a group-convoluted feature map; adding the down-sampled feature map and the group-convoluted feature map, and then performing activation through an activation layer to obtain a feature map of the other feature extraction module.

2. The head-mounted display device-based gesture recognition method of claim 1, wherein, the group convolution processing comprises: inputting the feature map into a group convolution module for multi-size kernel feature extraction; inputting the output feature of the multi-size kernel feature extraction into a normalization layer and an activation function layer to output the group-convoluted feature map.

3. The head-mounted display device-based gesture recognition method of claim 2, wherein, the inputting the output feature of the multi-size kernel feature extraction into the normalization layer and the activation function layer to output the group-convoluted feature map comprises: for a last feature extraction module in the plurality of stacked feature extraction modules: inputting the output feature of the multi-size kernel feature extraction into the normalization layer to obtain the group-convoluted feature map; adding the down-sampled feature map and the group-convoluted feature map, and then performing activation through an activation layer to obtain a feature map of the last feature extraction module.

4. The head-mounted display device-based gesture recognition method of claim 2, wherein, the inputting the feature map into the group convolution module for multi-size kernel feature extraction comprises: channel disassembling the feature map to obtain a plurality of groups of channels, the convolution kernel of each group of channels being different; performing group convolution in each group of channels, wherein the number of groups is equal to the number of channels; then, channel splicing the results of each group convolution; finally, transforming the feature map through an attention mechanism network.

5. The head-mounted display device-based gesture recognition method of claim 4, wherein, after the inputting the feature map into the group convolution module for multi-size kernel feature extraction, the method further comprises: channel scrambling to mix the features extracted from different size kernels.

6. The head-mounted display device-based gesture recognition method of claim 1, wherein, before the inputting the image into the hand key point detection model, the method further comprises: detecting a hand region image in the image by a hand detection model; tracking the hand region image by a tracker and inputting the tracked hand region image into the hand key point detection model.

7. The head-mounted display device-based gesture recognition method of claim 6, wherein, The hand detection model comprises: The hand detection model is obtained by changing or deleting the dimension unfolding layer and / or the fully connected layer in the hand key point detection model.

8. The head-mounted display device-based gesture recognition method of claim 7, wherein, The tracking of the hand region image by the tracker further comprises: Every N frames, the hand detection model is used again to detect a hand region image in a new image.

Images