Hand posture estimation method and system based on visual and inertial information fusion

Abstract

The invention discloses a hand posture estimation method and system based on visual and inertial information fusion. The method comprises the following steps: (1) constructing a hand posture data set; (2) extracting features: performing visual information feature extraction on a color image acquired by AR glasses through a Resnet50 residual network to finally obtain an image feature vector; performing inertial information feature extraction by constructing a convolutional neural network to obtain an inertial information feature vector; connecting the image feature vector and the inertial feature vector to obtain a fused feature vector; (3) carrying out hand 2D attitude estimation; (4) performing hand 3D posture estimation; (5) carrying out network training and testing; and (6) deploying the trained hand posture estimation network model to AR glasses, and performing real-time hand posture estimation by calling a color camera and a data glove.

Description

technical field [0001] The present invention relates to the fields of human-computer interaction, deep learning, and computer vision, in particular to a hand posture estimation method and system based on fusion of visual and inertial information. Background technique [0002] With the rapid development of the intelligent age, "human-centered" human-computer interaction has attracted widespread attention. Compared with the traditional interaction methods using mouse, keyboard, etc., the emerging human-computer interaction method that uses the natural properties of the human body breaks the restrictions on operating distance, single method, and rule setting, and becomes a more friendly human-computer interaction. direction of interaction research. Gesture is an auxiliary communication method widely used in people's daily life. The structure of the human hand is flexible, the movements are diverse, and it can express rich semantic information, and the gesture expression is nat...

AI Technical Summary

Problems solved by technology

[0004] Although the traditional single sensor can realize hand pose estimation in a specific scene, it is difficult to meet the complex actual interaction scene.

In order to promote the research and analysis of multi-modal hand pose estimation methods, more and more studies focus on combining visual information and wearable information. Vision-based devices can provide absolute measurement information of hand pose. Wearable-based devices The system can perform data supplementation when the vision-based camera equipment does not capture the hand image and hand occlusion, so as to perform more complex human-computer interaction work. In the prior art, the use of a single sensor for hand pose estimation has a good effect. However, due to the limitation of a single sensor itself, it is difficult to meet the human-computer interaction work in complex application scenarios, so it is of great theoretical research significance and practical application value to explore high-precision, strong immersive real-time gesture pose estimation based on multi-modal information

Images