Gesture recognition method of VR equipment and VR equipment

Abstract

The invention provides a gesture recognition method of VR (virtual reality) equipment and the VR equipment, which are used for improving the accuracy of gesture recognition during VR interaction. Comprising the steps that hand detection is conducted on a target hand image, a feature map of the target hand image and the position of a hand bounding box in the target hand image are obtained, and the feature map comprises hand image features and background image features in the target hand image; based on the position of the hand bounding box and the feature map of the target hand image, a hand feature map is obtained, and the hand feature map comprises hand image features; extracting local region features of the hand feature map to obtain a left hand feature vector and a right hand feature vector; inputting the left hand feature vector and the right hand feature vector into an encoder network to extract hand interaction features so as to obtain two-hand interaction features; and decoding the two-hand interaction feature by using a decoder network to obtain a gesture classification feature, and obtaining a gesture recognition result of the target hand image based on the gesture classification feature.

Description

Background Art

[0002] Gesture interaction is a very important part of the VR (Virtual Reality) interactive experience. Compared with using a handle to interact, gesture interaction is more convenient and direct, and can bring users a more intuitive immersive experience. At present, gesture interaction is mainly completed through 3D hand reconstruction and gesture recognition, among which gesture recognition is responsible for distinguishing specific gestures and providing operation feedback for related applications.

[0003] The gesture recognition method in the prior art usually uses a 3D gesture algorithm to perform three-dimensional reconstruction on each specific gesture, such as Figure 1 As shown, it is a schematic diagram of a three-dimensional model of a two-handed cross gesture, and Figure 2 The following is a schematic diagram of simulating a camera gesture with both hands. Then, gesture recognition is performed based on the reconstructed 3D model of each gesture. However, this method has high requirements on the accuracy of the 3D reconstruction model. Therefore, when the accuracy of the reconstructed 3D model corresponding to each gesture is low, the accuracy of gesture recognition will be low. Summary of the invention

[0004] The present application provides a gesture recognition method for a VR device and a VR device, which are used to improve the accuracy of gesture recognition during VR interaction.

[0005] In a first aspect, an embodiment of the present application provides a gesture recognition method for a VR device, the method comprising:

[0006] For any frame of the target hand image obtained by the VR device, perform hand detection on the target hand image to obtain a feature map of the target hand image and the position of the hand bounding box in the target hand image, wherein the feature map includes hand image features and background image features in the target hand image;

[0007] Based on the position of the hand bounding box and the feature map of the target hand image, a hand feature map is obtained, wherein the hand feature map includes the hand image features;

[0008] Performing local area feature extraction on the hand feature map to obtain a left hand feature vector and a right hand feature vector;

[0009] Inputting the left hand feature vector and the right hand feature vector into an encoder network to extract hand interaction features to obtain two-hand interaction features;

[0010] The two-hand interaction feature is decoded by using a decoder network to obtain a gesture classification feature, and a gesture recognition result of the target hand image is obtained based on the gesture classification feature.

[0011] A second aspect of the present application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0012] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0013] For any frame of the target hand image obtained by the VR device, perform hand detection on the target hand image to obtain a feature map of the target hand image and the position of the hand bounding box in the target hand image, wherein the feature map includes hand image features and background image features in the target hand image;

[0014] Based on the position of the hand bounding box and the feature map of the target hand image, a hand feature map is obtained, wherein the hand feature map includes the hand image features;

[0015] Performing local area feature extraction on the hand feature map to obtain a left hand feature vector and a right hand feature vector;

[0016] Inputting the left hand feature vector and the right hand feature vector into an encoder network to extract hand interaction features to obtain two-hand interaction features;

[0017] The two-hand interaction feature is decoded by using a decoder network to obtain a gesture classification feature, and a gesture recognition result of the target hand image is obtained based on the gesture classification feature.

[0018] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect.

[0019] In the above-mentioned embodiment of the present application, the position of the hand bounding box obtained by hand detection on the target hand image and the feature map of the target hand image are used to obtain a hand feature map. Local area feature extraction is performed on the hand feature map to obtain a left hand feature vector and a right hand feature vector. The left hand feature vector and the right hand feature vector are then input into the encoder network to extract hand interaction features to obtain two-hand interaction features. The two-hand interaction features are decoded using a decoder network to obtain gesture classification features. And based on the gesture classification features, the gesture recognition result of the target hand image is obtained. Therefore, in the embodiment of the present application, the target hand image can be directly gesture recognized through the neural network model, and there is no need to recognize the three-dimensional reconstruction model of each gesture, thereby avoiding the low accuracy of gesture recognition caused by the low precision of the three-dimensional model after the reconstruction of each gesture. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 One of the schematic diagrams of the two-handed gesture model provided in the embodiment of the present application is exemplarily shown;

[0022] Figure 2 The second schematic diagram of the two-handed gesture model provided in the embodiment of the present application is exemplarily shown;

[0023] Figure 3 One of the flowcharts of the gesture recognition method of the VR device provided in the embodiment of the present application is exemplarily shown;

[0024] Figure 4 The schematic diagram of the structure of the encoder network provided by the embodiment of the present application is exemplarily shown;

[0025] Figure 5 The structural diagram of the self-attention mechanism module provided in the embodiment of the present application is exemplarily shown;

[0026] Figure 6 The following is a schematic diagram showing an exemplary process of determining a target hand image gesture recognition result provided by an embodiment of the present application;

[0027] Figure 7 The schematic diagram of the structure of the network model of gesture recognition provided by the embodiment of the present application is exemplified;

[0028] Figure 8 The second flowchart of the gesture recognition method for a VR device provided in an embodiment of the present application is exemplarily shown;

[0029] Fig. 9 A schematic diagram of a gesture recognition device for a VR device provided in an embodiment of the present application is exemplarily shown;

[0030] Fig.10 The hardware structure diagram of the VR device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0031] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0032] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0033] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0034] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device that includes a series of components is not necessarily limited to those components that are clearly listed, but may include other components that are not clearly listed or inherent to these products or devices.

[0035] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0036] The idea of ​​the embodiments of the present application is summarized below.

[0037] At present, 3D gesture algorithms are usually used to reconstruct each specific gesture in three dimensions, and then gesture recognition is performed based on the reconstructed three-dimensional models of each gesture. However, this method has high requirements on the accuracy of the three-dimensional reconstruction model. Therefore, when the accuracy of the reconstructed three-dimensional model corresponding to each gesture is low, the accuracy of gesture recognition will be low.

[0038] Based on the prior art, there is a problem that the accuracy of gesture recognition is low due to the low precision of the reconstructed three-dimensional model corresponding to each gesture. The embodiment of the present application provides a gesture recognition method for a VR device, which obtains the position of the hand bounding box and the feature map of the target hand image by performing hand detection on the target hand image, and obtains a hand feature map. And the local area feature extraction is performed on the hand feature map to obtain the left hand feature vector and the right hand feature vector. Then the left hand feature vector and the right hand feature vector are input into the encoder network to extract the hand interaction feature to obtain the two-hand interaction feature. The two-hand interaction feature is decoded by the decoder network to obtain the gesture classification feature. And the gesture recognition result of the target hand image is obtained based on the gesture classification feature. Therefore, in the embodiment of the present application, the target hand image can be directly gesture recognized by the neural network model, and the three-dimensional reconstruction model of each gesture does not need to be recognized, which avoids the low accuracy of gesture recognition caused by the low precision of the three-dimensional model reconstructed by each gesture. Below, the gesture recognition method of the VR device in the embodiment of the present application is described in detail in conjunction with the accompanying drawings.

[0039] like Figure 3 FIG. 1 is a flow chart of a gesture recognition method for a VR device, which may include the following steps:

[0040] Step 301: for any frame of a target hand image obtained by a VR device, perform hand detection on the target hand image to obtain a feature map of the target hand image and the position of a hand bounding box in the target hand image, wherein the feature map includes hand image features and background image features in the target hand image;

[0041] The target hand image in the embodiment of the present application can be each frame of image captured by the VR device, or can be a hand image obtained at every specified number of frames among the hand images obtained by the VR device. The specific setting can be made according to the actual situation, and the embodiment of the present application does not limit the target hand image.

[0042] Secondly, the hand detection method in the embodiment of the present application is to input the target hand image into the hand detection algorithm for hand detection. The hand detection algorithm in the embodiment of the present application may be a hand detection algorithm composed of a Backbone module and / or a Head module, wherein the Backbone module is used to extract features of the target hand image to obtain a feature map of the target hand image, and the Head module is used to detect the hand position in the target hand image to obtain the position of the hand bounding box in the target hand image. However, if the target hand image is in a tracking state, the Head module may not be used for hand position detection, and the Kalman filter algorithm may be used to determine the position of the hand bounding box in the current frame target hand image based on the previous frame target hand image.

[0043] It should be noted that the hand detection algorithm described above is only used for illustration and does not limit the hand detection algorithm in the embodiments of the present application. The hand detection algorithm in the embodiments of the present application can be set according to actual conditions.

[0044] Step 302: obtaining a hand feature map based on the position of the hand bounding box and the feature map of the target hand image, wherein the hand feature map includes the hand image features;

[0045] The hand bounding box in the embodiment of the present application includes a left hand bounding box and / or a right hand bounding box, and the hand feature map includes a left hand feature map and / or a right hand feature map.

[0046] In one embodiment, step 302 can be specifically implemented as follows: cropping the feature map using the position of the left hand bounding box to obtain the left hand feature map; and / or cropping the feature map using the position of the right hand bounding box to obtain the right hand feature map.

[0047] Step 303: extracting local area features from the hand feature map to obtain a left hand feature vector and a right hand feature vector;

[0048] In one embodiment, the left hand feature vector and the right hand feature vector can be obtained by the following three methods:

[0049] Method 1: If the hand feature map includes the left hand feature map, the ROI Align algorithm is used to extract local area features of the left hand feature map to obtain the left hand local area features, and the left hand local area features are filtered to obtain a left hand feature vector of a specified size, and the preset feature vector is determined as the right hand feature vector.

[0050] Method 2: If the hand feature map includes the right hand feature map, the ROI Align algorithm is used to extract local area features of the right hand feature map to obtain the right hand local area features, and the right hand local area features are filtered to obtain a right hand feature vector of a specified size, and the preset feature vector is determined as the left hand feature vector.

[0051] Method three: If the hand feature map includes the left hand feature map and the right hand feature map, the ROI Align algorithm is used to extract local area features of the left hand feature map to obtain the left hand local area features, and the ROI Align algorithm is used to extract local area features of the right hand feature map to obtain the right hand local area features, and the left hand local area features and the right hand local area features are filtered respectively to obtain the left hand feature vector and the right hand feature vector.

[0052] The method of extracting local area features of the left-hand feature map and / or the right-hand feature map using the ROI Align algorithm in the embodiment of the present application is a method in the prior art, and the embodiment of the present application will not be repeated here. In addition, the method of filtering the left-hand local area features and / or the right-hand local area features in the embodiment of the present application is to use a 1*1 convolution layer or a 1*1 pooling layer to filter the left-hand local area features and / or the right-hand local area features. It can be set according to actual conditions, and the embodiment of the present application does not limit the filtering method here. And the preset vector in the embodiment of the present application is an all-zero vector whose size is equal to the specified size.

[0053] Step 304: inputting the left hand feature vector and the right hand feature vector into an encoder network to extract hand interaction features, thereby obtaining two-hand interaction features;

[0054] like Figure 4 As shown, it is a schematic diagram of the structure of the encoder network in the embodiment of the present application, from Figure 4 It can be seen that the encoder network 400 includes a first bias-free feedforward neural network 401, a second bias-free feedforward neural network 402, a self-attention mechanism module 403, a feature weighting module 404, and a feature fusion module 405. Next, the method of determining the two-handed interaction feature in step 304 is described in conjunction with the specific structure of the encoder network:

[0055] The first bias-free feedforward neural network 401 is used to extract the left hand feature vector to obtain the left hand feature weight; and the second bias-free feedforward neural network 402 is used to extract the right hand feature vector to obtain the right hand feature weight. The left hand feature vector and the right hand feature vector are respectively extracted by the self-attention mechanism module 403 to obtain the left hand interaction feature and the right hand interaction feature. The left hand feature vector is feature-weighted based on the left hand feature weight by the feature weighting module 404 to obtain the weighted left hand feature vector, and the right hand feature vector is feature-weighted based on the right hand feature weight by the feature weighting module 404 to obtain the weighted right hand feature vector. The weighted left hand feature vector and the weighted right hand feature vector are fused by the feature fusion module 405 to obtain the two-hand interaction feature.

[0056] In one embodiment, the feature weighting module 404 is used to perform feature weighting on the left-hand hand feature vector based on the left-hand hand feature weight to obtain the weighted left-hand hand feature vector. The specific implementation is: multiplying the left-hand hand feature weight with the left-hand hand feature vector to obtain the weighted left-hand hand feature vector.

[0057] In one embodiment, the feature weighting module 404 is used to perform feature weighting on the right hand feature vector based on the right hand feature weight to obtain the weighted right hand feature vector. The specific implementation is: multiplying the right hand feature weight by the right hand feature vector to obtain the weighted right hand feature vector.

[0058] In one embodiment, the feature fusion module 405 is used to fuse the weighted left hand feature vector and the weighted right hand feature vector to obtain the two-hand interaction feature. Specifically, the two-hand interaction feature is implemented as follows:

[0059] The weighted left hand feature vector and the weighted right hand feature vector are added to obtain the two-hand interaction feature.

[0060] like Figure 5 As shown, the self-attention mechanism module 403 in the embodiment of the present application includes a multi-head self-attention mechanism 4031 and a dual-head self-attention mechanism 4032. The specific manner of extracting features of the left hand feature vector and the right hand feature vector respectively through the self-attention mechanism module 403 is described below:

[0061] The multi-head self-attention mechanism 4031 is used to perform feature extraction on the left-hand feature vector and the right-hand feature vector respectively to obtain the intermediate left-hand interaction feature and the intermediate right-hand interaction feature; the dual-head self-attention mechanism 4032 is used to perform feature extraction on the intermediate left-hand interaction feature and the intermediate right-hand interaction feature respectively to obtain the left-hand interaction feature and the right-hand interaction feature.

[0062] Step 305: Decoding the two-hand interaction feature using a decoder network to obtain a gesture classification feature;

[0063] In one embodiment, step 305 may be specifically implemented as follows: inputting the two-hand gesture interaction feature and the gesture classification feature of the target hand image in the previous frame into the decoder network to obtain the gesture classification feature.

[0064] The decoder network in the embodiment of the present application is a network structure in the prior art. The decoder network in the embodiment of the present application may be a decoder network composed of LSTM (Long Short-Term Memory), but the structure of the decoder network is not limited in the embodiment of the present application. The decoder network in the embodiment of the present application may be set according to actual conditions.

[0065] Step 306: Obtaining a gesture recognition result of the target hand image based on the gesture classification feature.

[0066] like Figure 6 The above is a schematic diagram of the process of determining the target hand image gesture recognition result in step 306, which includes the following steps:

[0067] Step 601: inputting the gesture classification feature into a linear classifier to obtain an intermediate gesture recognition result, wherein the intermediate gesture recognition result includes a left-hand gesture recognition result, a right-hand gesture recognition result, and a two-hand gesture recognition result;

[0068] Step 602: Based on the gesture recognition result of the target hand image in the previous frame and the intermediate gesture recognition result, obtain the gesture recognition result of the target hand image in the current frame.

[0069] In one embodiment, step 602 can be specifically implemented in the following three ways:

[0070] Method 1: If the gesture recognition result of the target hand image in the previous frame is a left-hand gesture recognition result, then determine that the gesture recognition result of the target hand image in the current frame is the left-hand gesture recognition result.

[0071] Method 2: If the gesture recognition result of the target hand image in the previous frame is a right-hand gesture recognition result, then determine that the gesture recognition result of the target hand image in the current frame is the right-hand gesture recognition result.

[0072] Mode three: if the gesture recognition result of the previous frame of the hand image is a two-hand gesture recognition result, then determine that the gesture recognition result of the current hand image is the two-hand gesture recognition result.

[0073] Next, the gesture recognition method of the VR device in the embodiment of the present application is described in detail in combination with the structure of the network model in the embodiment of the present application. Figure 7 , which is a schematic diagram of the structure of a network model 700 for gesture recognition in an embodiment of the present application. The network model for gesture recognition includes a hand detection layer 701, an ROI layer 702, a convolutional layer 703, an encoder network 704, a decoder network 705, and a linear classifier 706. Among them:

[0074] For any frame of the target hand image obtained by the VR device, the hand detection layer 701 is used to perform hand detection on the target hand image to obtain the feature map of the target hand image and the position of the hand bounding box in the target hand image; and based on the position of the hand bounding box and the feature map of the target hand image, a hand feature map is obtained. Then, the ROI layer 702 is used to extract the local area features of the left hand feature map to obtain the local area features of the left hand, and the ROI layer 702 is used to extract the local area features of the right hand feature map to obtain the local area features of the right hand; and the convolution layer 703 is used to filter the local area features of the left hand and the local area features of the right hand, respectively, to obtain the left hand feature vector and the right hand feature vector. The left hand feature vector and the right hand feature vector are input into the encoder network 704 to extract hand interaction features to obtain two-hand interaction features, and then the two-hand interaction features are decoded by the decoder network 705 to obtain gesture classification features. Finally, the gesture recognition result of the target hand image is obtained based on the gesture classification features through the linear classifier 706.

[0075] In order to further connect the methods in this application, such as Figure 8 FIG. 1 is a flow chart of a gesture recognition method for a VR device in the present application, which may include the following steps:

[0076] Step 801: for any frame of a target hand image obtained by a VR device, perform hand detection on the target hand image to obtain a feature map of the target hand image and the position of a hand bounding box in the target hand image, wherein the feature map includes hand image features and background image features in the target hand image;

[0077] The hand bounding box in the embodiment of the present application includes a left hand bounding box and / or a right hand bounding box, and the hand feature map includes a left hand feature map and / or a right hand feature map.

[0078] Step 802: cropping the feature map using the position of the left hand bounding box to obtain the left hand feature map; and / or cropping the feature map using the position of the right hand bounding box to obtain the right hand feature map;

[0079] Step 803: If the hand feature map includes a left hand feature map, extract local area features from the left hand feature map using the ROI Align algorithm to obtain left hand local area features, filter the left hand local area features to obtain a left hand feature vector of a specified size, and determine the preset feature vector as the right hand feature vector;

[0080] Step 804: If the hand feature map includes a right hand feature map, extract local area features from the right hand feature map using the ROI Align algorithm to obtain right hand local area features, filter the right hand local area features to obtain a right hand feature vector of a specified size, and determine the preset feature vector as the left hand feature vector;

[0081] Step 805: If the hand feature map includes the left hand feature map and the right hand feature map, then use the ROIAlign algorithm to extract local area features of the left hand feature map to obtain left hand local area features, and use the ROI Align algorithm to extract local area features of the right hand feature map to obtain right hand local area features, and filter the left hand local area features and the right hand local area features respectively to obtain the left hand feature vector and the right hand feature vector;

[0082] Step 806: Input the left hand feature vector and the right hand feature vector into the encoder network, perform feature extraction on the left hand feature vector using a first unbiased feedforward neural network in the encoder network to obtain a left hand feature weight; and perform feature extraction on the right hand feature vector using a second unbiased feedforward neural network in the encoder network to obtain a right hand feature weight;

[0083] Step 807: extracting features of the left hand feature vector and the right hand feature vector respectively through the self-attention mechanism module in the encoder network to obtain left hand interaction features and right hand interaction features;

[0084] Step 808: using the left hand feature weight and the right hand feature weight to fuse the left hand interaction feature and the right hand interaction feature to obtain the two-hand interaction feature;

[0085] Step 809: Decoding the two-hand interaction feature using a decoder network to obtain a gesture classification feature;

[0086] Step 810: inputting the gesture classification feature into a linear classifier to obtain an intermediate gesture recognition result, wherein the intermediate gesture recognition result includes a left-hand gesture recognition result, a right-hand gesture recognition result, and a two-hand gesture recognition result;

[0087] Step 811: based on the gesture recognition result of the target hand image in the previous frame and the intermediate gesture recognition result, obtain the gesture recognition result of the target hand image in the current frame.

[0088] Based on the same inventive concept, the gesture recognition method of the VR device disclosed above can also be implemented by a gesture recognition device of the VR device. The effect of the gesture recognition device of the VR device is similar to that of the aforementioned method, and will not be repeated here.

[0089] Fig. 9 The figure is a schematic diagram of the structure of a gesture recognition device for a VR device according to an embodiment of the present disclosure.

[0090] like Fig. 9 As shown, the gesture recognition device 900 of the VR device of the present disclosure may include a hand detection module 910, a hand feature extraction module 920, a local area feature extraction module 930, a two-handed interaction feature extraction module 940 and a gesture recognition result determination module 950.

[0091] The hand detection module 910 is used to perform hand detection on any frame of a target hand image obtained by the VR device to obtain a feature map of the target hand image and the position of a hand bounding box in the target hand image, wherein the feature map includes hand image features and background image features in the target hand image;

[0092] A hand feature extraction module 920, configured to obtain a hand feature map based on the position of the hand bounding box and the feature map of the target hand image, wherein the hand feature map includes the hand image features;

[0093] A local area feature extraction module 930 is used to extract local area features from the hand feature map to obtain a left hand feature vector and a right hand feature vector;

[0094] A two-hand interaction feature extraction module 940 is used to input the left hand feature vector and the right hand feature vector into an encoder network to extract hand interaction features and obtain two-hand interaction features;

[0095] The gesture recognition result determination module 950 is used to decode the two-hand interaction feature by using a decoder network to obtain a gesture classification feature, and obtain a gesture recognition result of the target hand image based on the gesture classification feature.

[0096] In one embodiment, the hand bounding box includes a left hand bounding box and / or a right hand bounding box, and the hand feature map includes a left hand feature map and / or a right hand feature map;

[0097] The hand feature extraction module 920 is specifically used for:

[0098] The feature map is cropped using the position of the left hand bounding box to obtain the left hand feature map; and / or,

[0099] The feature map is cropped using the position of the right hand bounding box to obtain the right hand feature map.

[0100] In one embodiment, the local area feature extraction module 930 is specifically used to:

[0101] If the hand feature map includes the left hand feature map, the ROI Align algorithm is used to extract local area features of the left hand feature map to obtain left hand local area features, and the left hand local area features are filtered to obtain a left hand feature vector of a specified size, and a preset feature vector is determined as the right hand feature vector; or,

[0102] If the hand feature map includes the right hand feature map, the ROI Align algorithm is used to extract local area features of the right hand feature map to obtain right hand local area features, and the right hand local area features are filtered to obtain a right hand feature vector of a specified size, and a preset feature vector is determined as the left hand feature vector; or,

[0103] If the hand feature map includes the left hand feature map and the right hand feature map, the ROIAlign algorithm is used to perform local area feature extraction on the left hand feature map to obtain the left hand local area feature, and the ROI Align algorithm is used to perform local area feature extraction on the right hand feature map to obtain the right hand local area feature, and the left hand local area feature and the right hand local area feature are filtered respectively to obtain the left hand feature vector and the right hand feature vector.

[0104] In one embodiment, the encoder network includes a first bias-free feedforward neural network, a second bias-free feedforward neural network, and a self-attention mechanism module;

[0105] The two-hand interaction feature extraction module 940 is specifically used for:

[0106] Using the first bias-free feedforward neural network to extract features from the left hand feature vector to obtain left hand feature weights; and using the second bias-free feedforward neural network to extract features from the right hand feature vector to obtain right hand feature weights;

[0107] The left hand feature vector and the right hand feature vector are respectively extracted by a self-attention mechanism module to obtain a left hand interaction feature and a right hand interaction feature;

[0108] The left-hand interaction feature and the right-hand interaction feature are fused using the left-hand feature weight and the right-hand feature weight to obtain the two-hand interaction feature.

[0109] In one embodiment, the self-attention mechanism module includes a multi-head self-attention mechanism and a dual-head self-attention mechanism;

[0110] The two-hand interaction feature extraction module 940 is further used for:

[0111] Using the multi-head self-attention mechanism, feature extraction is performed on the left hand feature vector and the right hand feature vector respectively to obtain an intermediate left hand interaction feature and an intermediate right hand interaction feature;

[0112] The dual-headed self-attention mechanism is used to extract features of the intermediate left-hand interaction feature and the intermediate right-hand interaction feature respectively to obtain the left-hand interaction feature and the right-hand interaction feature.

[0113] In one embodiment, the gesture recognition result determination module 950 is specifically used to:

[0114] Inputting the gesture classification features into a linear classifier to obtain an intermediate gesture recognition result, wherein the intermediate gesture recognition result includes a left-hand gesture recognition result, a right-hand gesture recognition result, and a two-hand gesture recognition result;

[0115] Based on the gesture recognition result of the target hand image in the previous frame and the intermediate gesture recognition result, the gesture recognition result of the target hand image in the current frame is obtained.

[0116] In one embodiment, the gesture recognition result determination module 950 is further used to:

[0117] If the gesture recognition result of the target hand image in the previous frame is a left-hand gesture recognition result, then determining that the gesture recognition result of the target hand image in the current frame is the left-hand gesture recognition result; or,

[0118] If the gesture recognition result of the target hand image in the previous frame is a right-hand gesture recognition result, then determining that the gesture recognition result of the target hand image in the current frame is the right-hand gesture recognition result; or,

[0119] If the gesture recognition result of the previous frame of hand image is a two-hand gesture recognition result, it is determined that the gesture recognition result of the current hand image is the two-hand gesture recognition result.

[0120] After introducing a gesture recognition method and apparatus for a VR device according to an exemplary embodiment of the present invention, next, an electronic device according to another exemplary embodiment of the present invention is introduced.

[0121] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0122] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the gesture recognition method of the VR device according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 3 Steps 301-305 shown in .

[0123] Refer to the following Fig.10 The electronic device 1000 according to this embodiment of the present invention will be described. Fig.10 The electronic device 1000 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0124] like Fig.10 As shown, the electronic device 1000 is in the form of a general electronic device. The components of the electronic device 1000 may include but are not limited to: the at least one processor 1001, the at least one computer storage medium 1002, and a bus 1003 connecting different system components (including the computer storage medium 1002 and the processor 1001).

[0125] Bus 1003 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0126] The computer storage medium 1002 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1021 and / or cache storage media 1022 , and may further include read-only computer storage media (ROM) 1023 .

[0127] The computer storage medium 1002 may also include a program / utility 1025 having a set (at least one) of program modules 1024, such program modules 1024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0128] The electronic device 1000 may also communicate with one or more external devices 1004 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or communicate with any device that enables the electronic device 1000 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1005. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1006. As shown, the network adapter 1006 communicates with other modules for the electronic device 1000 via a bus 1003. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0129] In some possible implementations, various aspects of the gesture recognition method for a VR device provided by the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the gesture recognition method for a VR device according to various exemplary embodiments of the present invention described above in this specification.

[0130] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A gesture recognition method for a VR device, characterized in that: The method comprises: For any frame of the target hand image obtained by the VR device, perform hand detection on the target hand image to obtain a feature map of the target hand image and the position of the hand bounding box in the target hand image, wherein the feature map includes hand image features and background image features in the target hand image; Based on the position of the hand bounding box and the feature map of the target hand image, a hand feature map is obtained, wherein the hand feature map includes the hand image features; Performing local area feature extraction on the hand feature map to obtain a left hand feature vector and a right hand feature vector; Inputting the left hand feature vector and the right hand feature vector into an encoder network to extract hand interaction features to obtain two-hand interaction features; The two-hand interaction feature is decoded by using a decoder network to obtain a gesture classification feature, and a gesture recognition result of the target hand image is obtained based on the gesture classification feature.

2. The method according to claim 1, characterized in that The hand bounding box includes a left hand bounding box and / or a right hand bounding box, and the hand feature map includes a left hand feature map and / or a right hand feature map; The obtaining of the hand feature map based on the position of the hand bounding box and the feature map of the target hand image includes: The feature map is cropped using the position of the left hand bounding box to obtain the left hand feature map, and / or, The feature map is cropped using the position of the right hand bounding box to obtain the right hand feature map.

3. The method according to claim 2, characterized in that Performing local area feature extraction on the hand feature map to obtain a left hand feature vector and a right hand feature vector, including: If the hand feature map includes the left hand feature map, the ROI Align algorithm is used to extract local area features of the left hand feature map to obtain left hand local area features, and the left hand local area features are filtered to obtain a left hand feature vector of a specified size, and a preset feature vector is determined as the right hand feature vector; or, If the hand feature map includes the right hand feature map, the ROI Align algorithm is used to extract local area features of the right hand feature map to obtain right hand local area features, and the right hand local area features are filtered to obtain a right hand feature vector of a specified size, and a preset feature vector is determined as the left hand feature vector; or, If the hand feature map includes the left hand feature map and the right hand feature map, the ROI Align algorithm is used to perform local area feature extraction on the left hand feature map to obtain the left hand local area feature, and the RoI Align algorithm is used to perform local area feature extraction on the right hand feature map to obtain the right hand local area feature, and the left hand local area feature and the right hand local area feature are filtered respectively to obtain the left hand feature vector and the right hand feature vector.

4. The method according to claim 1, characterized in that: The encoder network includes a first bias-free feedforward neural network, a second bias-free feedforward neural network and a self-attention mechanism module; The step of inputting the left hand feature vector and the right hand feature vector into an encoder network to extract hand interaction features to obtain two-hand interaction features includes: Using the first bias-free feedforward neural network to extract features from the left hand feature vector to obtain left hand feature weights; and using the second bias-free feedforward neural network to extract features from the right hand feature vector to obtain right hand feature weights; Performing feature extraction on the left hand feature vector and the right hand feature vector respectively through a self-attention mechanism module to obtain a left hand interaction feature and a right hand interaction feature; The left-hand interaction feature and the right-hand interaction feature are fused using the left-hand feature weight and the right-hand feature weight to obtain the two-hand interaction feature.

5. The method according to claim 4, characterized in that The self-attention mechanism module includes a multi-head self-attention mechanism and a dual-head self-attention mechanism; The step of extracting features of the left hand feature vector and the right hand feature vector respectively through the self-attention mechanism module to obtain left hand interaction features and right hand interaction features includes: Using the multi-head self-attention mechanism, feature extraction is performed on the left hand feature vector and the right hand feature vector respectively to obtain an intermediate left hand interaction feature and an intermediate right hand interaction feature; The dual-headed self-attention mechanism is used to extract the intermediate left-hand interaction feature and the intermediate right-hand interaction feature respectively to obtain the left-hand interaction feature and the right-hand interaction feature.

6. The method according to claim 1, characterized in that The step of obtaining a gesture recognition result of the target hand image based on the gesture classification feature includes: Inputting the gesture classification features into a linear classifier to obtain an intermediate gesture recognition result, wherein the intermediate gesture recognition result includes a left-hand gesture recognition result, a right-hand gesture recognition result, and a two-hand gesture recognition result; Based on the gesture recognition result of the target hand image in the previous frame and the intermediate gesture recognition result, the gesture recognition result of the target hand image in the current frame is obtained.

7. The method according to claim 6, characterized in that The step of obtaining the gesture recognition result of the target hand image of the current frame based on the gesture recognition result of the target hand image of the previous frame and the intermediate gesture recognition result includes: If the gesture recognition result of the target hand image in the previous frame is a left-hand gesture recognition result, then determining that the gesture recognition result of the target hand image in the current frame is the left-hand gesture recognition result; or, If the gesture recognition result of the target hand image in the previous frame is a right-hand gesture recognition result, then determining that the gesture recognition result of the target hand image in the current frame is the right-hand gesture recognition result; or, If the gesture recognition result of the previous frame of hand image is a two-hand gesture recognition result, it is determined that the gesture recognition result of the current hand image is the two-hand gesture recognition result.

8. A VR device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: For any frame of the target hand image obtained by the VR device, perform hand detection on the target hand image to obtain a feature map of the target hand image and the position of the hand bounding box in the target hand image, wherein the feature map includes hand image features and background image features in the target hand image; Based on the position of the hand bounding box and the feature map of the target hand image, a hand feature map is obtained, wherein the hand feature map includes the hand image features; Performing local area feature extraction on the hand feature map to obtain a left hand feature vector and a right hand feature vector; Inputting the left hand feature vector and the right hand feature vector into an encoder network to extract hand interaction features to obtain two-hand interaction features; The two-hand interaction feature is decoded by using a decoder network to obtain a gesture classification feature, and a gesture recognition result of the target hand image is obtained based on the gesture classification feature.

9. The VR device according to claim 8, characterized in that: The hand bounding box includes a left hand bounding box and / or a right hand bounding box, and the hand feature map includes a left hand feature map and / or a right hand feature map; The processor executes the feature map based on the position of the hand bounding box and the target hand image to obtain the hand feature map, which is specifically configured as follows: The feature map is cropped using the position of the left hand bounding box to obtain the left hand feature map, and / or, The feature map is cropped using the position of the right hand bounding box to obtain the right hand feature map.

10. The VR device according to claim 9, characterized in that: The processor performs local area feature extraction on the hand feature map to obtain a left hand feature vector and a right hand feature vector, and is specifically configured as follows: If the hand feature map includes the left hand feature map, the ROI Align algorithm is used to extract local area features of the left hand feature map to obtain left hand local area features, and the left hand local area features are filtered to obtain a left hand feature vector of a specified size, and a preset feature vector is determined as the right hand feature vector; or, If the hand feature map includes the right hand feature map, the ROI Align algorithm is used to extract local area features of the right hand feature map to obtain right hand local area features, and the right hand local area features are filtered to obtain a right hand feature vector of a specified size, and a preset feature vector is determined as the left hand feature vector; or, If the hand feature map includes the left hand feature map and the right hand feature map, the ROI Align algorithm is used to perform local area feature extraction on the left hand feature map to obtain the left hand local area feature, and the ROI Align algorithm is used to perform local area feature extraction on the right hand feature map to obtain the right hand local area feature, and the left hand local area feature and the right hand local area feature are filtered respectively to obtain the left hand feature vector and the right hand feature vector.

Images