High-speed rail simulated driving action virtual and real interaction method based on gesture recognition

Abstract

The invention discloses a high-speed rail simulated driving action virtual-real interaction method based on gesture recognition, and the method specifically comprises the steps: fixing a depth camera on a head-mounted display, and enabling the three-dimensional data captured by the depth camera to be fused in a head-mounted display coordinate system; 21 hand joint points of a hand are detected by using a MediaPipe framework, and three-dimensional coordinates of the hand joint points are obtained by using depth data; drawing a hand model in a Unity 3D platform by taking the three-dimensional coordinates of the 21 hand joint points as base points, and updating in real time according to the change of the coordinates; and converting the real object model into a head-mounted display coordinate system, and realizing virtual-real interaction of the simulated driving action according to the position relationship between the hand model and the real object model. According to the method, the real object and the hand coordinates are fused and mapped to the virtual environment, interaction with the real object is achieved through the gesture action in the virtual environment, and the method has the advantages of being flexible in operation, resistant to shielding interference and the like.

Description

technical field [0001] The invention belongs to the technical field of virtual reality (VR) / augmented reality (AR) and high-speed rail simulation driving, in particular to a virtual-real interaction method for high-speed rail simulation driving actions based on gesture recognition. Background technique [0002] In the virtual reality environment, the coordinates of objects in the real and virtual environments are independent of each other, and it is difficult to interact with objects in the real environment through what is seen in the virtual environment; current virtual reality applications can use handles, data gloves, and machine vision. interact. Interaction by using the handle limits the movement of the hand and cannot interact with real objects; the data glove can interact with real objects, but the data glove also restricts the movement of the hand and is expensive, and the sensors in the data glove cannot provide location information. It is usually necessary to inst...

AI Technical Summary

Problems solved by technology

Using the handle to interact restricts hand movements and cannot interact with real objects; data gloves can interact with real objects, but data gloves also restrict hand movements and are expensive. At the same time, sensors in data gloves cannot provide location information. It is usually necessary to install an additional position tracking system on the glove; the use of computer vision for 3D hand pose estimation not only retains the comfort of bare-hand interaction, but also relatively cheap, which is a major direction for the development of virtual reality interaction methods

Images