AR scene-based gesture interaction method and device, storage medium, and communication terminal

Abstract

The present disclosure relates to the field of augmented reality technology, and in particular to an AR scene-based gesture interaction method, an AR scene-based gesture interaction device, a computer-readable medium, and a communication terminal. The method includes: collecting the RGB image, the depth image of the current frame, and the corresponding IMU data; processing the RGB image, the depth image and the IMU data to obtain the posture information and hand information of the current frame; according to the posture information of the current frame Perform 3D dense modeling with hand information to obtain 3D point cloud information of the hand in the reference coordinate system; and obtain pose information of virtual objects in the reference coordinate system; based on the 3D point cloud of the hand in the reference coordinate system information, the pose information of the virtual object, and the preset point cloud information of the virtual object to render the hand and the virtual object, so as to obtain the occlusion relationship between the hand and the virtual object. The present disclosure can make the hand and the virtual object have a more accurate contact display effect.

Description

technical field [0001] The present disclosure relates to the field of augmented reality technology, and in particular to an AR scene-based gesture interaction method, an AR scene-based gesture interaction device, a computer-readable medium, and a communication terminal. Background technique [0002] Augmented reality (AR) is a method of applying virtual information to the real world through information technology, and superimposing the real environment and virtual objects on the same picture or space in real time. With the rapid development of smart mobile terminals, interaction with AR gestures can be realized on terminal devices. In an existing solution, simple interaction with virtual objects in an AR scene can be realized through a 2D SLAM model based on the gesture recognition result of a 2D image. In another solution, based on the gesture recognition result of the 3D image, the interaction with the virtual object in the AR scene can be realized through the 2D SLAM mod...

AI Technical Summary

Problems solved by technology

[0003] However, there are also certain shortcomings and deficiencies in the prior art. For example, most of the solutions in the prior art drive the actions or behaviors of virtual objects based on gesture recognition results.

However, the occlusion relationship between the hand and the virtual object in the AR scene cannot be realized, and the "contact" with the virtual object cannot be realized, which leads to the unrealistic interactive experience between the user and the virtual object in the AR scene.

Images