Positioning tracking and virtuality and reality combination method
Through the Kinect camera group and KinectFusion algorithm, three-dimensional model and spatial coordinates are established, combined with dynamic texture mapping, the problems of untimely detection of users' limbs and poor visual effects in virtual reality technology are solved, real life-like virtual images and fast interactions are achieved Experience.
Patent Information
- Application Number
- CN202410996554.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
AI Technical Summary
In the current virtual reality technology, in car driving training, users' limb detection response is not timely, and the three-dimensional model has poor visual effects, affecting the interactive experience.
The Kinect camera group is used to collect video images of the user's upper limbs, lower limbs and operating objects in real time, and three-dimensional reconstruction is carried out through the KinectFusion algorithm, three-dimensional model diagrams and spatial coordinates are established, and dynamic texture mapping is combined with virtual scenes to judge user actions and reconstruct virtual scenes.
It improves the fidelity and three-dimensionality of virtual images, reduces delays, and improves the interactive experience of user driving simulation.
Smart Images

Figure CN118941746A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of virtual reality, and in particular relates to a method for positioning, tracking and combining virtuality and reality. Background Art
[0002] In recent years, virtual reality (VR) technology and VR interactive devices have been increasingly widely used in virtual training and virtual maintenance of large equipment. At home and abroad, there have been a number of VR-based aircraft virtual maintenance systems, engineering machinery virtual operation training systems, and combat vehicle driving simulation and control systems. For automobile driving training, the sensors on the simulator detect the movements of the steering wheel, gear lever, etc. and transmit them to the display device. The display device does not respond in time, and the user's limbs often use a general three-dimensional model, which has poor visual effects and affects the user's operation and interactive experience. Summary of the invention
[0003] The technical problem to be solved by the present invention is to provide a method for positioning tracking and combining virtuality and reality, which can provide users with realistic, three-dimensional, and timely responsive virtual images, thereby improving the user's interactive experience of driving simulation.
[0004] In order to solve the above technical problems, the technical solution of the present invention is:
[0005] A method for positioning, tracking and combining virtuality and reality, comprising:
[0006] The Kinect camera group collects video images of the user's upper limbs, lower limbs and the operated object in real time, and obtains depth images and color images from different perspectives;
[0007] The KinectFusion algorithm is used to perform 3D reconstruction on the depth images of different viewing angles to obtain the overall 3D scene model, and the overall 3D scene model is imported into the Meshlab software in the obj format, and the Meshlab software is used to segment it to establish a 3D model map of the user's upper limbs, lower limbs and operation objects consistent with the video image, as well as the 3D spatial coordinates before and after the action;
[0008] The display unit establishes a virtual scene according to the virtual scene information, and fuses the real-time three-dimensional spatial coordinates and the three-dimensional model diagram of the user's upper limbs, lower limbs and the operation object with the virtual scene through a dynamic texture mapping method to generate a corresponding virtual-real fusion image;
[0009] It is determined whether the virtual three-dimensional objects in the virtual scene and the effective actions of the operation object meet the preset prerequisites according to the preset events. If so, the corresponding scene information is obtained from the stored virtual content information to reconstruct the virtual scene.
[0010] Furthermore, the virtual scene includes an in-vehicle virtual scene and a road scene, and the road virtual scene is constructed and transformed accordingly according to the virtual three-dimensional objects in the road scene and the valid actions of the operation object state.
[0011] Furthermore, the operation objects include physical operation objects and virtual operation objects. The physical operation objects include pedals, a steering wheel and a gear lever. The virtual operation objects include buttons, levers or knobs on and near the steering wheel.
[0012] Furthermore, finger feature information is extracted from the captured video image to obtain the three-dimensional spatial coordinates of the user's finger. When the three-dimensional spatial coordinates of the user's finger coincide with the three-dimensional spatial coordinates of the virtual operation object and there is a pressing, toggling or rotating action, the virtual operation object undergoes an effective action.
[0013] Furthermore, the video image is matched with the image information and depth information of a standard physical operation object, and the real-time three-dimensional spatial coordinates of the physical operation object are calculated. When the three-dimensional spatial coordinates of the physical operation object change, the physical operation object performs an effective action.
[0014] Furthermore, the display unit is a head-mounted display device, and the Kinect camera group is installed on the display device and / or installed independently.
[0015] After adopting the above technical solution, the beneficial effects of the present invention are:
[0016] Since the positioning tracking and virtual-reality integration method of the present invention tracks and locates by collecting video images of the user's upper limbs, lower limbs and operating objects, and establishes a three-dimensional model diagram and three-dimensional space coordinates, which can be integrated into the virtual scene, so as to display a realistic and three-dimensional virtual image in the display unit, the processing and transmission speed of the image information is fast, so that the conversion of character actions and road scenes in the virtual fused image is timely, the delay is reduced, and the user's interactive experience of driving simulation is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a flow chart of the positioning tracking and virtual-real combination method of the present invention. DETAILED DESCRIPTION
[0018] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the scope of protection of the present invention.
[0019] like Figure 1As shown, a method for positioning tracking and combining virtuality and reality includes the following steps:
[0020] S1, Kinect camera group collects video images of the user's upper limbs, lower limbs and operated objects in real time to obtain depth images and color images from different perspectives;
[0021] S2. Use KinectFusion algorithm to perform 3D reconstruction on depth images of different perspectives to obtain the overall 3D scene model. Import the overall 3D scene model into Meshlab software in obj format, segment it using Meshlab software, and establish a 3D model of the user's upper limbs, lower limbs and operation objects consistent with the video image, as well as the 3D spatial coordinates before and after the action;
[0022] S3, the display unit establishes a virtual scene according to the virtual scene information, and fuses the real-time three-dimensional spatial coordinates and three-dimensional model diagram of the user's upper limbs, lower limbs and operation objects with the virtual scene through a dynamic texture mapping method to generate a corresponding virtual-real fusion image;
[0023] S4. Determine whether the effective actions of the virtual three-dimensional objects and operation objects in the virtual scene meet the preset prerequisites according to the preset events. If so, obtain the corresponding scene information from the stored virtual content information to reconstruct the virtual scene.
[0024] In step S3, the virtual scene includes an in-car virtual scene and a road scene, and the road virtual scene is constructed and transformed accordingly according to the virtual three-dimensional objects in the road scene and the valid actions of the operation object state.
[0025] Specifically, the operation objects include physical operation objects and virtual operation objects. The physical operation objects include pedals, a steering wheel, and a gear lever. The virtual operation objects include buttons, levers, or knobs on and near the steering wheel.
[0026] The specific method for judging whether the virtual operation object has an action is to extract the finger feature information of the collected video image to obtain the three-dimensional spatial coordinates of the user's finger. When the three-dimensional spatial coordinates of the user's finger coincide with the three-dimensional spatial coordinates of the virtual operation object and there is a pressing, sliding or rotating action, the virtual operation object has an effective action.
[0027] The specific method for judging whether the physical operation object has an action is to match the video image with the image information and depth information of the standard physical operation object, calculate the real-time three-dimensional spatial coordinates of the physical operation object, and when the three-dimensional spatial coordinates of the physical operation object change, the physical operation object has an effective action.
[0028] In step S4, the corresponding scene information is obtained from the stored virtual content information to construct the virtual scene, specifically to reconstruct the road scene, and the virtual states of the virtual vehicle such as turning, shifting, accelerating and decelerating are realized through the transformation of the road scene.
[0029] Furthermore, the display unit is a head-mounted display device, and the Kinect camera group is installed on the display device and / or installed independently. Specifically, the Kinect cameras are all installed on the surrounding side of the display device, or the Kinect camera for shooting upper limbs is installed on the surrounding side of the display device, and can capture the movements of structures such as the upper limbs, gear lever, steering wheel and nearby buttons. The Kinect camera for shooting lower limbs is independently installed on a fixed structure near the pedals, ensuring that the user's lower limbs and pedal movements can be captured.
[0030] The positioning tracking and virtual-reality integration method of the present invention tracks and locates the user by collecting video images of the user's upper limbs, lower limbs and operating objects, and establishes a three-dimensional model diagram and three-dimensional space coordinates, which can be integrated into the virtual scene, thereby displaying a realistic and three-dimensional virtual image in the display unit. The image information is processed and transmitted at a fast speed, so that the character actions and road scenes in the virtual fused image are converted in time, which reduces delays, thereby improving the user's interactive experience of driving simulation.
[0031] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the described embodiments are only some embodiments of the present invention, rather than all embodiments, which are merely examples, and the protection scope of the present invention is defined by the claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principle and essence of the present invention and without any creative work, but these changes and modifications all fall within the protection scope of the present invention.
Claims
1. A method for positioning, tracking and combining virtuality and reality, characterized in that: include: The Kinect camera group collects video images of the user's upper limbs, lower limbs and the operated object in real time, and obtains depth images and color images from different perspectives; The KinectFusion algorithm is used to perform 3D reconstruction on the depth images of different viewing angles to obtain the overall 3D scene model, and the overall 3D scene model is imported into the Meshlab software in the obj format, and the Meshlab software is used to segment it to establish a 3D model map of the user's upper limbs, lower limbs and operation objects consistent with the video image, as well as the 3D spatial coordinates before and after the action; The display unit establishes a virtual scene according to the virtual scene information, and fuses the real-time three-dimensional spatial coordinates and the three-dimensional model diagram of the user's upper limbs, lower limbs and the operation object with the virtual scene through a dynamic texture mapping method to generate a corresponding virtual-real fusion image; It is determined whether the virtual three-dimensional objects in the virtual scene and the effective actions of the operation object meet the preset prerequisites according to the preset events. If so, the corresponding scene information is obtained from the stored virtual content information to reconstruct the virtual scene.
2. The method for positioning tracking and combining virtuality and reality according to claim 1, characterized in that: The virtual scene includes an in-vehicle virtual scene and a road scene, and the road virtual scene is constructed and transformed accordingly according to the virtual three-dimensional objects in the road scene and the valid actions of the operation object state.
3. The method for positioning, tracking and combining virtuality and reality according to claim 2, characterized in that: The operation objects include physical operation objects and virtual operation objects. The physical operation objects include pedals, a steering wheel and a gear lever. The virtual operation objects include buttons, levers or knobs on and near the steering wheel.
4. The method for positioning, tracking and combining virtuality and reality according to claim 3, characterized in that: Finger feature information is extracted from the collected video image to obtain the three-dimensional spatial coordinates of the user's finger. When the three-dimensional spatial coordinates of the user's finger coincide with the three-dimensional spatial coordinates of the virtual operation object and there is a pressing, toggling or rotating action, the virtual operation object performs an effective action.
5. The method for positioning, tracking and combining virtuality and reality according to claim 3, characterized in that: The video image is matched with the image information and depth information of a standard physical operation object, and the real-time three-dimensional spatial coordinates of the physical operation object are calculated. When the three-dimensional spatial coordinates of the physical operation object change, the physical operation object performs an effective action.
6. The method for positioning, tracking and combining virtuality and reality according to any one of claims 1 to 5, characterized in that: The display unit is a head-mounted display device, and the Kinect camera group is installed on the display device and / or installed independently.
