VR handle positioning method and equipment

Abstract

The invention relates to the technical field of VR, and provides a VR handle positioning method and equipment, a VR handle performs own 3Dof pose calculation, nonlinear 6Dof pose optimization calculation and 6Dof pose prediction calculation, and VR equipment performs exposure imaging of a near-infrared LED lamp ring, LED light spot detection, construction of a 2D-3D mapping relation between LED light spots and LED lamps, and 6Dof initial pose calculation of the VR handle. The VR handle only needs to transmit the pose calculation result of the VR equipment at the required moment to the VR equipment in a low-frequency manner, the pressure of transmission bandwidth is reduced, and when the problems of electromagnetic interference, channel occupation, frequency band occupation and the like exist, the VR handle can ensure that data is not lost in a data retransmission manner, so that the positioning precision and robustness of the VR handle are improved, and meanwhile, the positioning accuracy and robustness of the VR handle are improved. Through cooperative calculation of the two devices, the positioning efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology and provides a VR handle positioning method and device. Background Art

[0002] Generally, an all-in-one VR machine consists of two parts: a VR device and a VR handle. As a key function of an all-in-one VR machine, handle positioning allows users to subjectively experience the three-dimensional interaction process between their hands and the virtual scene.

[0003] At present, computer vision and inertial navigation technology are mainly used for handle positioning. By deploying the handle positioning algorithm on the VR device side, the VR handle sends the high-frequency data (such as 500Hz) collected by the inertial measurement unit (IMU) to the VR device through wireless communication. The system on chip (System on Chip) of the VR device uses the handle positioning algorithm to calculate the 6 degrees of freedom (Dof) of the handle based on the IMU data of the VR handle and the LED light ring image on the VR handle collected by the camera. However, when the communication frequency band or channel of the VR device and the VR handle is occupied, or there is electromagnetic interference in the use environment, data frame loss may occur during the positioning process. However, due to the limitation of the wireless communication bandwidth, the high-frequency IMU data of the handle cannot be retransmitted on a large scale to ensure that no frame loss occurs, especially when the data of two VR handles need to be transmitted at the same time. In this way, the posture calculation on the VR device side will be wrong, resulting in inaccurate or failed positioning results. Summary of the invention

[0004] The embodiments of the present application provide a VR handle positioning method and device for improving the robustness of handle positioning.

[0005] On the one hand, an embodiment of the present application provides a VR handle positioning algorithm, which is applied to a VR handle, wherein the VR handle is synchronized with the time axis of a VR device, and the method includes:

[0006] Calculate the 3D pose of the VR controller in real time based on the motion data continuously collected by the built-in IMU;

[0007] Receive an exposure instruction sent by the VR device according to the camera exposure frequency, and light up the LED light ring according to the current exposure time indicated by the exposure instruction;

[0008] Repeating sending a first message in response to the exposure instruction to the VR device until receiving feedback from the VR device on the first message; wherein the first message includes a 3Dof pose of the VR handle at the current exposure moment and a 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment, and the 3Dof pose and the 6Dof predicted pose are used to determine a 2D-3D mapping relationship between an LED light spot and an LED light;

[0009] Receive a data packet repeatedly sent by the VR device before calculating the next 6DOf real pose of the VR handle in the world coordinate system, the data packet including a 2D-3D mapping relationship between LED light spots and LED lights, a 2D coordinate of each LED light spot in a handle image collected at the current exposure time, and a 6DOf initial pose of the VR handle in the world coordinate system at the current exposure time;

[0010] According to the received data packet and the motion data collected by the built-in IMU before the data packet is received, nonlinearly optimize the 6Dof initial posture of the VR handle in the world coordinate system to obtain the 6Dof target posture;

[0011] Determine the 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment according to the 6Dof target pose;

[0012] Receive a rendering instruction sent by the VR device, and send a second message in response to the rendering instruction to the VR device, wherein the second message carries the 6Dof predicted posture, so that the VR device renders and displays the VR handle in the current screen according to the 6Dof predicted posture.

[0013] On the other hand, an embodiment of the present application provides a VR handle positioning method, which is applied to a VR device, wherein the VR device is synchronized with the time axis of the VR handle, and the method includes:

[0014] Sending an exposure command to the VR handle according to the camera exposure frequency, obtaining a handle image captured by the camera when the LED light ring is on at the current exposure moment, and performing LED spot detection on the handle image to obtain the 2D coordinates of each LED spot;

[0015] Receive a first message sent by the VR handle in response to the exposure instruction, and if the handle image is a first frame exposure image, determine the 2D-3D mapping relationship between the LED light spot and the LED light according to the 3Dof pose of the VR handle at the current exposure moment contained in the first message; otherwise, determine the 2D-3D mapping relationship between the LED light spot and the LED light according to the 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment contained in the first message;

[0016] Determine the 6Dof initial position of the VR handle in the world coordinate system at the current exposure moment according to the 2D-3D mapping relationship between the LED light spots and the LED lights and the 2D coordinates of each LED light spot;

[0017] Encapsulate the 2D-3D mapping relationship between the LED light spots and the LED lights, the 2D coordinates of each LED light spot, and the 6Dof initial pose of the VR handle into a data packet, and repeatedly send it to the VR handle before calculating the next 6Dof initial pose of the VR handle in the world coordinate system;

[0018] Sending a rendering instruction to the VR controller according to the rendering frequency to obtain a 6Dof predicted position of the VR controller in the world coordinate system at the current rendering moment indicated by the rendering instruction;

[0019] A second message is received in which the VR controller replies to the exposure instruction, and the VR controller is rendered and displayed in the current screen according to the 6Dof predicted posture of the VR controller at the current rendering moment carried in the second message.

[0020] On the other hand, an embodiment of the present application provides a VR handle, which is synchronized with the time axis of a VR device, and the VR handle includes a near-infrared LED light ring, an IMU, a memory, a processor, and a communication interface, and the near-infrared LED light ring, the IMU, the memory, and the processor are connected via a bus;

[0021] The memory stores a computer program, and the processing performs the following operations according to the computer program:

[0022] Calculate the 3D pose of the device in real time based on the motion data continuously collected by the IMU;

[0023] Receiving, via the communication interface, an exposure instruction sent by the VR device according to the camera exposure frequency, and lighting up the near-infrared LED light ring according to the current exposure time indicated by the exposure instruction;

[0024] Repeating, through the communication interface, sending a first message in response to the exposure instruction to the VR device until receiving feedback from the VR device on the first message; wherein the first message includes a 3Dof pose of the VR handle at the current exposure moment and a 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment, and the 3Dof pose and the 6Dof predicted pose are used to determine a 2D-3D mapping relationship between an LED light spot and an LED light;

[0025] Receiving, through the communication interface, a data packet repeatedly sent by the VR device before calculating the next 6DOf real pose, the data packet including a 2D-3D mapping relationship between LED light spots and LED lights, a 2D coordinate of each LED light spot in a handle image collected at the current exposure moment, and a 6DOf initial pose of the VR handle in the world coordinate system at the current exposure moment;

[0026] Through the communication interface, according to the received data packet and the motion data collected by the IMU before the data packet is received, nonlinearly optimize the 6Dof initial posture of the VR handle in the world coordinate system to obtain the 6Dof target posture;

[0027] Determine the 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment according to the 6Dof target pose;

[0028] Through the communication interface, a rendering instruction sent by the VR device is received, and a second message in response to the rendering instruction is sent to the VR device, wherein the second message carries the 6Dof predicted posture, so that the VR device renders and displays the VR handle in the current screen according to the 6Dof predicted posture.

[0029] On the other hand, an embodiment of the present application provides a VR device, wherein the time axis of the VR device and the VR handle are synchronized, and the VR device includes a plurality of near-infrared cameras, a display, a memory, a processor, and a communication interface;

[0030] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0031] Through the communication interface, an exposure instruction is sent to the VR handle according to the exposure frequency of the near-infrared camera, an image of the handle when the LED light ring is on at the current exposure moment is obtained, and LED spot detection is performed on the handle image to obtain the 2D coordinates of each LED spot;

[0032] Receiving, through the communication interface, a first message sent by the VR handle in response to the exposure instruction, if the handle image is a first-frame exposure image, determining a 2D-3D mapping relationship between the LED light spot and the LED light according to the 3Dof pose of the VR handle at the current exposure moment contained in the first message; otherwise, determining a 2D-3D mapping relationship between the LED light spot and the LED light according to the 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment contained in the first message;

[0033] Determine the 6Dof initial position of the VR handle in the world coordinate system at the current exposure moment according to the 2D-3D mapping relationship between the LED light spots and the LED lights and the 2D coordinates of each LED light spot;

[0034] Before the next 6Dof initial posture calculation, the 2D-3D mapping relationship between the LED light spot and the LED light, the 2D coordinates of each LED light spot and the 6Dof initial posture of the VR handle are packaged into a data packet and repeatedly sent to the VR handle through the communication interface;

[0035] Sending a rendering instruction to the VR controller through the communication interface according to the rendering frequency of the display;

[0036] Through the communication interface, a second message in which the VR handle replies to the exposure instruction is received, and the VR handle is rendered in the current screen according to the 6Dof predicted posture of the VR handle at the current rendering moment carried in the second message, and displayed through the display.

[0037] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a VR handle positioning method provided in an embodiment of the present application.

[0038] In a VR handle positioning method and device provided in an embodiment of the present application, the VR handle and the VR device are synchronized on the time axis, and the 6Dof posture of the VR handle is jointly positioned through the interaction between the VR device and the VR handle. During the positioning process, the VR handle calculates its own 3Dof posture in real time based on the motion data continuously collected by the built-in IMU, and lights up the LED light ring after receiving the exposure command sent by the VR device. At this time, the camera of the VR device collects the handle image when the LED light ring is lit, and determines the 2D coordinates of each LED spot. In response to the exposure command, the VR handle carries the 3Dof posture of the VR handle at the current exposure moment or the 6Dof predicted posture of the VR handle in the world coordinate system in the first message and sends it repeatedly to the VR device to prevent data loss. The VR device determines the 2D-3D mapping relationship between the LED light spot and the LED light according to the first message, and obtains the 3D coordinates of the LED light corresponding to each LED light spot according to the 2D-3D mapping relationship, and determines the 6Dof initial posture of the VR handle in the world coordinate system at the current exposure moment in combination with the 2D coordinates of the corresponding LED light spot, and packages the 2D-3D mapping relationship, the 2D coordinates of each LED light spot and the 6Dof initial posture of the VR handle and repeatedly sends them to the VR handle to prevent data loss. The VR handle combines the motion data collected by the IMU before receiving the data packet to perform nonlinear optimization on the 6Dof initial posture of the VR handle. When receiving the rendering instruction sent by the VR device, the 6Dof predicted posture of the VR handle in the world coordinate system at the current rendering moment determined according to the optimized 6Dof target posture is sent to the VR device through the second message. The VR device renders and displays the VR handle in the current screen according to the 6Dof predicted posture of the VR handle at the current rendering moment to reflect the three-dimensional interaction between the user's hand and the virtual scene. By having the VR handle and the VR device jointly execute the handle positioning algorithm, the computing resource consumption on the VR device side is reduced. During the positioning process, the VR device only needs to send the current data to the VR device at a low frequency, so that the influence of electromagnetic interference, channel occupancy, frequency band occupancy and other problems on the posture calculation can be avoided through data retransmission, thereby improving the positioning accuracy and robustness of the VR handle, reducing the pressure on the transmission bandwidth and improving positioning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A schematic diagram of a VR all-in-one device provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0042] Figure 3 This is an overall architecture diagram of the VR handle positioning method provided in an embodiment of the present application;

[0043] Figure 4 This is a flowchart of the implementation of the VR handle end in the VR handle positioning algorithm provided in the embodiment of the present application;

[0044] Figure 5 This is the format of a single message sent from a VR controller to a VR device.

[0045] Figure 6 This is the implementation flow chart of the VR device side in the VR handle positioning algorithm;

[0046] Figure 7 A flow chart of a method for establishing a 2D-3D mapping relationship between an LED light spot and an LED lamp;

[0047] Figure 8 A flow chart of another method for establishing a 2D-3D mapping relationship between an LED light spot and an LED lamp;

[0048] Fig. 9 This is the format of a single message sent from a VR device to a VR controller.

[0049] Fig.10 Schematic diagram of the communication mechanism between the VR device and the VR controller when not affected by the usage environment;

[0050] Fig.11 Schematic diagram of the communication mechanism between the VR device and the VR controller when affected by the usage environment;

[0051] Fig.12 This is a flow chart of the interaction method between VR device and VR controller;

[0052] Fig.13 This is the structural diagram of the VR controller;

[0053] Fig.14 The structure of the VR device. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0055] like Figure 1 As shown, it is a schematic diagram of the VR all-in-one provided for the implementation of this application. The VR all-in-one generally includes a VR device and a VR handle. Among them, the VR device includes multiple near-infrared cameras, each camera can be uniquely identified by a camera serial number (such as Arabic numerals 1-4), and an IMU is installed inside the VR device to collect the motion data of the head. Through pre-calibration, the internal parameters of each camera and the external parameters between the camera and the IMU built into the VR device can be obtained. Each VR handle has an annular fixed shell, and the shell material has a high transmittance to infrared light. A number of near-infrared LED light sources are encapsulated under the shell to form an LED light ring with a fixed 3D structure. The 3D coordinates of each near-infrared LED light on the LED light ring are known, and each near-infrared LED light is provided with a unique identifier. An IMU is also installed inside the VR handle to collect the motion data of the handle.

[0056] Usually, before an all-in-one VR machine leaves the factory, the time axis of the VR device and the VR controller are synchronized so that the timestamps of the SOC systems of the two devices at the same time remain aligned. After the time axis is synchronized, the VR controller can clearly know the exposure time of the next frame of the controller image collected by the VR device's camera and the rendering time of the next frame of the controller image. The VR controller also locally stores the pre-calibrated VR device camera intrinsic parameters and the extrinsic parameters between the camera and the VR device's built-in IMU, so that the VR controller can subsequently calculate the reprojection error when performing nonlinear optimization of the 6Dof pose.

[0057] like Figure 2 As shown in the figure, it is a schematic diagram of the application scenario of the VR all-in-one machine. The user operates the VR handle with his hand to control the virtual image of the VR device. Therefore, the handle positioning is a key function of the VR all-in-one machine. Its positioning accuracy directly affects the accuracy of virtual image manipulation. Through the rendering display of the handle, the user can subjectively experience the three-dimensional interaction process between his hand and the virtual scene.

[0058] At present, the handle positioning algorithm is mainly deployed on the VR device side. After the positioning program is started, the VR device uses the camera to image the LED light ring on the VR handle, and continuously receives the motion data collected at high frequency from the built-in IMU of the VR handle, thereby calculating the 6Dof position of the handle. Since all-in-one VR machines are widely used in environments that include Bluetooth devices, WiFi devices, and wireless devices (such as shopping malls, conference venues, multi-device interactions, etc.), when the VR handle sends the motion data collected by the built-in IMU to the VR device at high frequency through 2.4GHz wireless communication technology, there may be electromagnetic interference, frequency band occupancy, channel occupancy and other problems, causing the IMU data of the VR handle to lose frames, resulting in the VR device being unable to effectively track the VR handle, resulting in VR handle positioning errors or failures.

[0059] In order to solve the impact of electromagnetic interference, frequency band occupancy, channel occupancy and other problems on the positioning of the VR handle during the use of the VR all-in-one machine, an embodiment of the present application provides a VR handle positioning algorithm, in which the VR handle performs 3Dof pose calculation, nonlinear 6Dof pose optimization calculation and 6Dof pose prediction calculation based on the motion data collected by the built-in IMU, and the VR device performs handle image acquisition, light spot detection, 2D-3D mapping relationship calculation between LED light spots and LED lights, 6Dof initial pose calculation and other processes, to implement a collaborative positioning algorithm based on two distributed platforms of the VR handle and the VR device, thereby reducing the consumption of computing resources on the VR device side, and during the positioning process, the VR handle does not need to transmit the motion data collected by the built-in IMU to the VR device at high frequency through wireless communication technology, but only needs to transmit the pose calculation results to the VR device at low frequency and repeatedly, thereby preventing the impact of data loss on the 6Dof pose calculation of the VR handle, improving positioning accuracy and robustness, while reducing the pressure on transmission bandwidth and improving positioning efficiency.

[0060] See also Figure 3, which is an overall architecture diagram of the VR handle positioning method provided in an embodiment of the present application, wherein the VR device has a built-in IMU and a near-infrared camera, the VR handle has a built-in IMU and a near-infrared LED light ring, the time axis of the VR device and the VR handle are synchronized, the near-infrared camera on the VR device is used to expose and image the near-infrared LED light ring on the VR handle, and the IMU includes an accelerometer and an angular accelerometer, which can measure 3-axis acceleration and 3-axis angular velocity. The SLAM system of the VR device can output its own 6Dof pose based on the 3-axis acceleration and 3-axis angular velocity of the head movement collected by the built-in IMU and the image collected by the near-infrared camera, combined with the calibration parameters of the IMU and the camera. The VR handle calculates its own 3Dof pose in real time based on the 3-axis acceleration and 3-axis angular velocity of the hand movement collected by the built-in IMU, and transmits the 3Dof pose required for the near-infrared camera exposure moment to the VR device. When the LED light ring is on, the VR device exposes and images the VR handle, and determines the area of ​​the VR handle in the exposure image based on the 6DOf predicted pose at the current exposure moment sent by the VR handle, thereby detecting the LED spot in the area and obtaining the 2D coordinates of each LED spot. The VR device establishes a mapping relationship between the two-dimensional LED spot and the three-dimensional LED light based on the 2D coordinates of each LED spot, the 3Dof pose of the VR handle, or the 6DOf predicted pose of the VR handle, and calculates the 6Dof initial pose of the VR handle using the PnP (n≥3) algorithm based on the mapping relationship and the 2D coordinates of each spot. The VR device packages the 6Dof initial pose of the VR handle, the 2D coordinates of each LED spot, and the mapping relationship and sends them to the VR handle. The VR handle combines the motion data collected by the IMU, uses the Ceres mathematical library to perform nonlinear optimization on the 6Dof initial pose, obtains the 6Dof target pose of the VR handle at the current exposure moment, and predicts based on the 6Dof target pose, and sends the prediction result to the VR device through 2.4GHz wireless communication technology.

[0061] The following describes the specific process of the VR handle positioning algorithm from the VR handle side and the VR device side respectively.

[0062] like Figure 4 As shown in the figure, it is the implementation flow chart of the VR handle end in the VR handle positioning algorithm. The process mainly includes the following steps:

[0063] S401: Calculate the 3D pose of the VR controller in real time based on the motion data continuously collected by the built-in IMU.

[0064] In one example, the motion data includes 3-axis acceleration and 3-axis angular velocity, and the 3Dof posture of the VR handle can be calculated in real time by using algorithms such as complementary filtering or Kalman filtering. The calculation result is a quaternion, and the 3D rotation posture can be obtained through the conversion relationship between the quaternion and the rotation matrix.

[0065] S402: Receive an exposure instruction sent by the VR device according to the camera exposure frequency, and light up the LED light ring according to the current exposure time indicated by the exposure instruction.

[0066] In order to save energy, the LED light ring on the VR handle lights up in a flashing manner, which requires the near-infrared camera on the VR device to expose at the same time as the near-infrared LED light ring on the VR handle lights up. Therefore, the VR device sends an exposure instruction to the VR handle according to the exposure frequency of the near-infrared camera (the exposure frequency of multiple cameras on the VR device is the same, such as 25Hz, that is, it is sent once every 40ms). The exposure instruction carries the current exposure time information and LED light ring lighting information. When the VR handle receives the exposure instruction, when the timestamp of the current exposure time arrives, the LED light ring is turned on so that the camera of the VR device exposes and images the LED light ring on the VR handle, thereby calculating the 6Dof pose of the handle in the world coordinate system based on the handle image at the current exposure time.

[0067] S403: Repeat sending the first message in response to the exposure instruction to the VR device until receiving feedback from the VR device in response to the first message.

[0068] The VR controller can obtain the current exposure moment of the camera on the VR device according to the received exposure instruction, and send the 3Dof posture of the VR controller at the current exposure moment to the VR device by carrying it in the first message.

[0069] Since the time axis of the VR device and the VR controller have been synchronized, when the camera on the VR device is not exposing the LED light ring on the VR controller for the first time, the VR controller can obtain the exposure time of the next frame of the controller image, and predict the 6Dof position of the VR controller in the world coordinate system at the exposure time of the next frame of the controller image based on the 6Dof position of the VR controller in the world coordinate system determined by the previous frame of the controller image. Therefore, for the first exposure imaging of the field, the VR controller can also carry the 6Dof predicted position of the VR controller in the world coordinate system at the current exposure time when sending the first message to the VR device.

[0070] Among them, the 3Dof posture and 6Dof predicted posture in the first message are used to determine the 2D-3D mapping relationship between the LED light spot and the LED light. For the specific determination process, please refer to the implementation on the VR device side.

[0071] In one example, in order to prevent data loss, the VR handle repeatedly sends the first message to the VR device through wireless communication technology before the next exposure moment until receiving feedback from the VR device on the first message, so as to ensure that the VR device receives the 3Dof pose and 6Dof predicted pose.

[0072] Compared with the camera technology in which the VR handle transmits the motion data collected by the built-in IMU to the VR device at a high frequency, in the embodiment of the present application, the VR handle only needs to transmit the posture calculation results (3Dof posture) and posture prediction results (6Dof posture) at the exposure moment to the VR device at a low frequency, which reduces the pressure on the transmission bandwidth and is less affected by electromagnetic interference, channel occupancy, and frequency band occupancy. In addition, since the transmission frequency of the posture results is low, the data loss problem can be solved by repeated transmission, which helps to improve positioning accuracy and robustness.

[0073] S404: Receive the data packet repeatedly sent by the VR device before calculating the next 6Dof real position of the VR handle in the world coordinate system.

[0074] In one example, the calculation process of the 6Dof pose of the VR handle in the world coordinate system is deployed on the VR device side. The VR device calculates the 6Dof initial pose of the VR handle in the world coordinate system based on the 3Dof pose and 6Dof predicted pose in the first message sent by the VR handle in response to the exposure command, combined with the handle image when the LED light ring is lit at the current exposure moment, and the 6Dof pose of the VR handle in the camera coordinate system output by the Simultaneous Localization and Mapping (SLAM) system of the VR device.

[0075] After calculating the 6DOF initial pose of the VR handle in the world coordinate system at the current exposure moment, the VR device encapsulates the 2D-3D mapping relationship between the LED light spots and the LED lights obtained in the calculation process, the 2D coordinates of each LED light spot in the handle image collected at the current exposure moment, and the 6DOF initial pose of the VR handle in the world coordinate system at the current exposure moment into a data packet, and sends it to the VR handle through wireless communication technology, so that the VR handle can perform nonlinear optimization on the 6DOF initial pose of the VR handle in combination with the motion data collected by the built-in IMU before receiving the data packet.

[0076] In one example, in order to ensure the effective transmission of the 6Dof initial pose corresponding to the current frame handle image and prevent data loss caused by electromagnetic interference, channel occupancy, frequency band occupancy and other problems, the VR device will repeatedly send the data packet corresponding to the current exposure moment to the VR handle before the VR device calculates the next 6Dof real pose of the VR handle in the world coordinate system.

[0077] S405: Perform nonlinear optimization on the 6Dof initial position of the VR controller in the world coordinate system according to the received data packet and the motion data collected by the built-in IMU before the data packet is received.

[0078] In one example, a VR handle receives a data packet from a VR device, and compares the data packet with the motion data collected by the built-in IMU before receiving the data packet (i.e., between the current exposure moment and the data packet reception moment), and uses the open source Ceres math library to calculate the reprojection error of each LED light spot on the handle image, as well as the IMU integral error for nonlinear optimization, to obtain the intrinsic parameters of the built-in IMU of the VR handle (including the bias of the accelerometer and angular velocity meter in three axes, the scale factor and the misalignment matrix (i.e., the rotation matrix for converting the accelerometer coordinate system to the angular velocity meter coordinate system) and other parameters) and the optimized 6Dof target pose of the VR handle in the world coordinate system.

[0079] S406: Determine the 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment according to the optimized 6Dof target pose.

[0080] In one example, after the VR handle obtains the optimized 6Dof target pose, a deep learning algorithm is used to determine the 6Dof predicted pose of the VR handle in the world coordinate system at the current rendering moment based on the optimized 6Dof target pose of the most recent exposure (usually the current exposure moment).

[0081] S407: Receive a rendering instruction sent by the VR device.

[0082] In order to allow users to more intuitively experience the interaction between their hands and the three-dimensional virtual scene, the three-dimensional model can be rendered and displayed in the virtual screen according to the 6Dof real posture of the VR handle. Therefore, in one example, the VR device can send a rendering instruction to the VR handle to request the 6Dof predicted posture of the VR handle at the current rendering moment according to the rendering frequency of the built-in display (such as 72Hz, that is, once every 13.9ms). Among them, the rendering instruction contains the current rendering moment information.

[0083] S408: Send a second message in response to the rendering instruction to the VR device, where the second message carries the 6Dof predicted posture of the VR controller at the current rendering moment, so that the VR device renders and displays the VR controller in the current screen according to the 6Dof predicted posture of the VR controller at the current rendering moment.

[0084] After receiving the rendering instruction, the VR handle sends the 6Dof predicted posture of the VR handle in the world coordinate system at the current rendering moment to the VR device through the second message. In this way, the VR device can use a filtering algorithm or an interpolation algorithm to perform smoothing operations on the 6Dof predicted posture of the VR handle at the current rendering moment and the 6Dof predicted posture of the VR handle at the previous rendering moment to obtain the final 6Dof predicted posture of the VR handle at the current rendering moment, and then render the VR handle according to the final 6Dof predicted posture and display it in the current picture, so that the user can intuitively feel the three-dimensional interaction process between the hand and the virtual scene.

[0085] In one example, the VR handle can also predict the 6Dof pose at the exposure moment and send the prediction result to the VR device to improve the calculation efficiency of the 6Dof pose at the next exposure moment. Specifically, the VR handle obtains the 6Dof target pose after nonlinear optimization of its 6Dof initial pose in the world coordinate system at the current exposure moment, and uses a deep learning algorithm to determine the 6Dof predicted pose of the VR handle in the world coordinate system at the next exposure moment according to the 6Dof target pose of the VR handle in the world coordinate system at the current exposure moment. In order to prevent data loss, before the next exposure moment arrives, the VR handle repeatedly sends messages to the VR device to transmit the 6Dof predicted pose at the next exposure moment. In this way, when the camera captures the handle image at the next exposure moment, the VR device can estimate the area of ​​the VR handle in the handle image based on the 6Dof predicted pose at the next exposure moment, realize rapid detection of the VR handle, and thus improve the calculation efficiency of the 6Dof real pose of the VR handle in the world coordinate system at the next exposure moment.

[0086] In one example, after the VR handle synchronizes the timeline with the VR device, the VR handle can obtain the next exposure moment of the VR device as well as the next rendering moment of the VR device, thereby predicting the 6Dof pose of the VR handle in the world coordinate system at the next rendering moment, and sending the prediction result to the VR device, so as to smooth the 6Dof pose of the VR handle and reduce the jitter of the VR handle in the virtual screen. Specifically, after the VR handle determines the 6Dof predicted pose of the VR handle in the world coordinate system at the current rendering moment, it uses a deep learning algorithm to determine the 6Dof predicted pose of the VR handle in the world coordinate system at the next rendering moment based on its 6Dof pose in the world coordinate system at the current rendering moment. In order to prevent data loss, before the next rendering moment arrives, the VR handle repeatedly sends messages to the VR device to transmit the 6Dof predicted pose of the next rendering moment, so as to smooth the 6Dof pose of the VR handle.

[0087] In the embodiment of the present application, the format of a single message sent by the VR handle to the VR device each time is as follows: Figure 5As shown in the figure, in addition to the message header and message footer, it also contains Part1-Part9 fields.

[0088] Part1 is the length of this message;

[0089] Part 2 is the timestamp of the last received LED light ring data, where the LED light ring data includes the 2D coordinates of each LED light spot in the most recently exposed handle image, the 2D-3D mapping relationship between the LED light spot and the LED light, etc.

[0090] Part3 is the timestamp of the 3Dof pose of the VR controller;

[0091] Part4 is the 3Dof pose;

[0092] Part5 is the timestamp of the current exposure moment;

[0093] Part 6 is the 6Dof predicted position of the VR controller in the world coordinate system at the current exposure moment;

[0094] Part7 is the timestamp of the current rendering moment;

[0095] Part 8 is the 6Dof predicted position of the VR controller in the world coordinate system at the current rendering moment;

[0096] Part9 is the verification code of the content of this message.

[0097] like Figure 6 As shown in the figure, it is the implementation flow chart of the VR device side in the VR handle positioning algorithm. The process mainly includes the following steps:

[0098] S601: Send an exposure command to the VR handle according to the camera exposure frequency, obtain the handle image captured by the camera when the LED light ring is on at the current exposure moment, and perform LED spot detection on the handle image to obtain the 2D coordinates of each LED spot.

[0099] The LED light ring on the VR handle lights up in a flashing manner. In order to ensure that the camera of the VR device can capture the LED light ring, the VR device can send exposure instructions to the VR handle according to the camera exposure frequency to notify the VR handle to light up the LED light ring at the current exposure time indicated by the exposure instruction, and perform infrared imaging when the LED light ring is on to obtain the handle image.

[0100] Furthermore, the VR device uses a deep learning algorithm or a traditional clustering algorithm to recognize the handle image at the current exposure moment, segment the handle area, and perform LED spot detection in the handle area to obtain the 2D coordinates of each LED spot in the handle image.

[0101] S602: Receive a first message sent by the VR controller in response to an exposure instruction.

[0102] Among them, the first message is sent repeatedly by the VR controller at a low frequency, including the 3Dof posture of the VR controller at the current exposure moment and the 6Dof predicted posture of the VR controller in the world coordinate system at the current exposure moment.

[0103] S603: Determine whether the handle image is the first exposure frame image, if so, execute S604, otherwise, execute S605.

[0104] The 3D structure of the LED light ring on the VR handle is known, but which LED light spot in the handle image corresponds to which LED light in the LED light ring on the VR handle is unknown. Depending on whether the handle image is the first frame exposure image, different methods can be used to establish a 2D-3D mapping relationship between the two-dimensional LED light spot and the three-dimensional LED light.

[0105] In one example, the 2D-3D mapping relationship between the LED light spot and the LED light can be expressed by the 2D coordinates of the LED light spot and the identifier of the LED light, such as {ID: (x, y)}, where ID represents the identifier of the LED light in the LED light ring, and (x, y) represents the 2D coordinates of the LED light spot formed by the LED light in the handle image.

[0106] S604: Determine the 2D-3D mapping relationship between the LED light spot and the LED light according to the 3Dof posture of the VR handle at the current exposure moment.

[0107] When the handle image is the first exposure frame, the process of determining the 2D-3D mapping relationship between the LED spot and the LED light can be found in Figure 7 , mainly includes the following steps:

[0108] S6041: Determine a 2D-3D mapping relationship between multiple groups of LED light spots and LED lights according to the 2D coordinates of each LED light spot and the 3D structure of the LED light ring.

[0109] In one example, the VR device analyzes the 3D structure of the LED light ring based on the 2D coordinates of each LED light spot in the handle image, determines the LED light that each LED light spot may correspond to through an enumeration algorithm, and obtains the 2D-3D mapping relationship between multiple groups of LED light spots and LED lights.

[0110] For each 2D-3D mapping relationship between a group of LED light spots and LED lights, execute steps S6042 to S6044.

[0111] S6042: Obtain the 3D coordinates of the LED lights corresponding to each LED spot, combine the 2D coordinates of each LED spot and the pre-calibrated camera internal parameters, and determine the 6DoF posture of the VR handle in the camera coordinate system.

[0112] In the specific implementation, according to the 2D-3D mapping relationship between the LED light spot and the LED light, the identification of the LED light corresponding to each LED light spot is determined, and according to each identification, the 3D coordinates of the LED light corresponding to the corresponding LED light spot are obtained. Then, the PnP or P3P algorithm is used to calculate the 6DoF posture of the VR handle in the camera coordinate system at the current exposure moment under the group of 2D-3D mapping relationships according to the 2D coordinates of each LED light spot and the 3D coordinates of the corresponding LED light. The posture includes a 3D coordinate representing the position of the VR handle in the camera coordinate system and a 3D rotation posture represented by a quaternion.

[0113] S6043: Obtain the 6DOf pose of the VR device in the world coordinate system at the current exposure moment according to its own 6DOf pose in the camera coordinate system, the pre-calibrated external parameters between the camera and the built-in IMU.

[0114] In the SLAM system, the world coordinate system is generally the coordinate system of the first frame of IMU data, and the external parameters between the camera and the built-in IMU on the VR device can be obtained through pre-calibration. Therefore, the VR device can determine the 6Dof pose of the VR device in the world coordinate system at the current exposure moment by using the external parameters between the camera and the built-in IMU through the head motion data collected by its own SLAM system and the built-in IMU. Specifically, the VR device multiplies its own 6Dof pose in the camera coordinate system by the external parameters of the camera and the built-in IMU on the VR device to obtain the 6Dof pose of the VR device in the world coordinate system at the current exposure moment.

[0115] S6044: Determine the 6DOf initial pose of the VR handle in the world coordinate system at the current exposure moment according to the 6DOf pose of the VR device in the world coordinate system and the 6DOf pose of the VR handle in the camera coordinate system at the current exposure moment.

[0116] In one example, the VR device multiplies its 6Dof posture in the world coordinate system at the current exposure moment by the 6Dof posture of the VR handle in the camera coordinate system at the current exposure moment to obtain the 6Dof real posture of the VR handle in the world coordinate system at the current exposure moment under this set of 2D-3D mapping relationships.

[0117] S6045: Use the 3Dof pose of the VR controller to verify the 6Dof initial poses corresponding to the multiple groups of 2D-3D mapping relationships, and select a group of 2D-3D mapping relationships.

[0118] Since the 6Dof initial posture of the VR handle in the world coordinate system at the current exposure moment includes a 3D coordinate representing the position of the VR handle in the world coordinate system and a 3D rotation posture represented by a quaternion, and the first message sent by the VR handle to the VR device includes a 3D rotation posture represented by a quaternion, the 3Dof posture at the current exposure moment contained in the first message can be used to verify the 6Dof initial posture corresponding to each group of 2D-3D mapping relationships, and a group of 2D-3D mapping relationships corresponding to the 6Dof initial posture with the smallest 3D rotation posture error at the current exposure moment is selected as the final 2D-3D mapping relationship. Specifically, the conversion relationship between quaternions and rotation matrices is used to convert the 3D rotation posture of the VR handle in the world coordinate system at the current exposure moment into a 3*3 first rotation matrix. At the same time, the 3D rotation posture in the 3DoF posture of the VR handle at the current exposure moment in the first message is also converted into a 3*3 second rotation matrix. The third row of the first rotation matrix is ​​extracted to obtain a 3*1 first vector, and the third row of the second rotation matrix is ​​extracted to obtain a 3*1 second vector. The first vector and the second vector are dot-multiplied. If the dot product result is closer to 1, the 2D-3D mapping relationship corresponding to the 6DoF initial posture is more accurate. Therefore, a set of 2D-3D mapping relationships closest to 1 is selected from multiple sets of calculation results as the final 2D-3D mapping relationship.

[0119] S605: Determine a 2D-3D mapping relationship between the LED light spot and the LED light according to the 6Dof predicted posture of the VR handle in the world coordinate system.

[0120] When the handle image is a non-frame exposure image, the process of determining the 2D-3D mapping relationship between the LED spot and the LED light can be found in Figure 8 , mainly includes the following steps:

[0121] S6051: Based on the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure moment and the pre-calibrated camera internal parameters, the LED light ring is projected onto the controller image at the current exposure moment.

[0122] In one example, the 3D structure of the LED light ring of the VR handle is known, and the 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment is also known. Using the internal parameters of the camera, the three-dimensional LED light ring can be projected into a two-dimensional handle image.

[0123] S6052: directly determining a 2D-3D mapping relationship between a group of LED light spots and LED lights according to the projection coordinates of each LED light spot in the handle image and the 2D coordinates of each LED light spot in the handle image.

[0124] In one example, for each LED light spot, the LED light corresponding to the projection coordinates closest to its 2D coordinates in the handle image is used as the LED light corresponding to the LED light spot, and a 2D-3D mapping relationship between the LED light spot and the LED light is obtained.

[0125] Relative to Figure 7 The 2D-3D mapping relationship determination process shown reduces the verification and screening process of the 2D-3D mapping relationship and improves the efficiency of pose calculation.

[0126] In one example, when the error between the projection coordinates of the LED light and the real 2D coordinates of the LED light spot is large, the determined mapping relationship may be inaccurate. Figure 7 The method shown determines the 2D-3D mapping relationship between the LED light spot and the LED light.

[0127] S606: Determine the 6Dof initial position of the VR handle in the world coordinate system at the current exposure moment according to the 2D-3D mapping relationship between the LED light spots and the LED lights and the 2D coordinates of each LED light spot.

[0128] During the specific implementation, the VR device obtains the 3D coordinates of the LED lights corresponding to each LED light spot based on the 2D-3D mapping relationship between the LED light spots and the LED lights. Combined with the 2D coordinates of each LED light spot, the PnP algorithm is used to calculate the 6DoF initial posture of the VR handle in the world coordinate system at the current exposure moment.

[0129] S607: Encapsulate the 2D-3D mapping relationship between the LED light spot and the LED light, the 2D coordinates of each LED light spot at the current exposure moment, and the 6Dof initial posture of the VR handle in the world coordinate system into a data packet, and repeatedly send it to the VR handle before calculating the next 6Dof initial posture of the VR handle in the world coordinate system.

[0130] After calculating the 6Dof initial pose of the VR handle in the world coordinate system at the current exposure moment, the VR device encapsulates the 2D-3D mapping relationship between the LED light spots and the LED lights obtained during the calculation process, the 2D coordinates of each LED light spot in the handle image collected at the current exposure moment, and the 6DOf initial pose of the VR handle in the world coordinate system at the current exposure moment into a data packet. In order to prevent data loss, before calculating the next 6Dof initial pose of the VR handle in the world coordinate system, the VR device repeatedly sends the data packet to the VR handle through wireless communication technology, so that the VR handle combines the motion data collected by the built-in IMU to perform nonlinear optimization on the 6Dof initial pose of the VR handle.

[0131] S608: Send a rendering instruction to the VR controller according to the rendering frequency to obtain the 6Dof predicted position of the VR controller in the world coordinate system at the current rendering moment indicated by the rendering instruction.

[0132] In order to allow users to more intuitively experience the interaction process between their hands and the three-dimensional virtual scene, the three-dimensional model can be rendered and displayed in the virtual screen according to the 6Dof posture of the VR controller at the current rendering moment. Therefore, in one example, the VR device can send a rendering instruction to the VR controller to request the 6Dof posture of the VR controller at the current rendering moment according to the rendering frequency of the built-in display (such as 72Hz, that is, once every 13.9ms).

[0133] S609: Receive a second message sent by the VR controller in response to the rendering instruction, and render and display the VR controller in the current screen according to the 6Dof predicted posture of the VR controller at the current rendering moment carried in the second message.

[0134] When the VR controller receives the rendering instruction, it estimates the 6Dof predicted posture of the VR controller in the world coordinate system at the current rendering moment according to the 6Dof target posture of the VR controller in the world coordinate system after optimization at the current exposure moment, and transmits the 6Dof predicted posture of the VR controller in the world coordinate system at the current rendering moment to the VR controller through the second message. The VR device renders the VR controller in the current screen through the graphics rendering engine according to the 6Dof predicted posture of the VR controller in the world coordinate system at the current rendering moment, so that the user can intuitively experience the three-dimensional interaction process between the hand and the virtual scene.

[0135] In the embodiment of the present application, the format of a single instruction sent by the VR device to the VR controller each time is as follows: Fig. 9 As shown in the figure, in addition to the message header and message footer, it also contains Part1-Part13 fields. Among them:

[0136] Part1 is the length of this message;

[0137] Part2 is a sign indicating whether the LED light ring is on, wherein when the LED is on (ie, Part is "yes"), Part3 is reset to "no".

[0138] Part 3 is a sign of whether the 3Dof pose of the VR controller is received during the LED light ring lighting cycle;

[0139] Part4 is the timestamp of the current exposure moment;

[0140] Part5 is the timestamp of the current rendering moment;

[0141] Part 6 is the timestamp for calculating the 6Dof real pose of the VR controller at the most recent exposure moment;

[0142] Part7 is the 6Dof real pose of the VR controller at the last exposure moment. If the 6Dof real pose of the VR controller is not calculated, the value of this field is marked as 0;

[0143] Part8 is the first serial number of the camera that observes the most LED spots. If the VR handle is not observed by any camera, the value of this field is set to 0;

[0144] Part9 is the number of LED spots observed by the camera with the first serial number;

[0145] Part10 is the second serial number of the camera that observed the second most LED spots. If the VR handle is not observed by any camera or is only observed by one camera, the value of this field is marked as 0;

[0146] Part11 is the number of LED light spots observed by the second camera;

[0147] Part12 is a variable-length field that describes the 2D coordinates of each LED light spot observed by the camera with the first sequence number and the camera with the second sequence number in sequence. If no LED light spot is observed by each camera, this field is empty.

[0148] Part 13 is the verification code of the content of this message.

[0149] In the absence of electromagnetic interference, signal occupancy, frequency band occupancy, etc., the uplink and downlink communication mechanism between the VR controller and the VR device is as follows: Fig.10 As shown in the figure, each time the VR device sends a command, the VR controller returns a message. Under the influence of electromagnetic interference, signal occupancy, frequency band occupancy, etc., the uplink and downlink communication mechanism between the VR device and the VR controller is as follows Fig.11 As shown. Among them:

[0150] The dotted line indicates the exposure command sent to the VR handle when the VR device exposes and images the LED light ring of the VR handle according to a fixed camera exposure frequency. The exposure command means to indicate the timestamp of the camera exposure imaging of the VR device to the VR handle and instruct the VR handle to light up the LED light ring. The contents of each field in the exposure command are shown in Table 1.

[0151] Table 1. Contents of the Part1-Part13 fields in the exposure command issued by the VR device

[0152] Part 1 L1 Part8 0 Part 2 yes Part9 0 Part 3 no Part10 0 Part 4 Timestamp T0 of the current exposure moment Part11 0 Part5 0 Part12 null Part6 0 Part13 Check code 1 Part7 0

[0153] After receiving the exposure command, the VR handle immediately replies with a message (indicated by a solid line). The message includes the current exposure time when the LED light ring indicated by the exposure command is lit, the 3Dof pose of the VR handle calculated based on the motion data collected by the built-in IMU, and the 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure time (if a prediction result exists). The contents of the Part1-Part9 fields in the message are shown in Table 2.

[0154] Table 2. Contents of the Part1-Part9 fields in the response message corresponding to the exposure command

[0155]

[0156] After the VR device receives the reply message sent by the VR controller to the exposure command, it compares the timestamp of the camera exposure imaging in the exposure command sent. If the timestamp of the 3Dof posture in the reply message is consistent with the timestamp in the exposure command, it replies with a confirmation message (indicated by a single dotted line) to notify the VR controller that it has received the data returned by the VR controller. The contents of the Part1-Part13 fields in the confirmation message sent by the VR device are shown in Table 3.

[0157] Table 3. Contents of the Part1-Part13 fields in the confirmation message sent by the VR device

[0158] Part 1 L3 Part8 0 Part 2 no Part9 0 Part 3 yes Part10 0 Part 4 Timestamp T0 of the current exposure moment Part11 0 Part5 0 Part12 null Part6 0 Part13 Check code 3 Part7 0

[0159] The double-dotted line indicates the rendering instruction sent by the VR device to the VR controller according to the rendering frequency of the display. The rendering instruction includes the timestamp of the current rendering moment to request the 6Dof predicted position of the VR controller in the world coordinate system at the current rendering moment. The contents of the Part1-Part13 fields in the rendering instruction are shown in Table 4.

[0160] Table 4. Contents of the Part1-Part13 fields in the rendering instructions issued by the VR device

[0161] Part 1 L4 Part8 0 Part 2 no Part9 0 Part 3 yes Part10 0 Part 4 0 Part11 0 Part5 Timestamp T1 of the current rendering moment Part12 null Part6 0 Part13 Check code 4 Part7 0

[0162] After receiving the rendering command sent by the VR device, the VR handle immediately replies with a message (indicated by a solid line) to send the 6Dof predicted position of the VR handle in the world coordinate system at the current rendering moment. The contents of the Part1-Part9 fields in the reply message to the rendering command are shown in Table 5.

[0163] Table 5. Contents of the Part1-Part9 fields in the reply message corresponding to the rendering command

[0164]

[0165] The long and short dashed lines represent the data packets sent by the VR device to the VR handle, including the 6Dof initial pose of the VR handle in the world coordinate system calculated based on the handle image collected at the current exposure moment, the 2D coordinates of each LED spot in the handle image, and the 2D-3D mapping relationship between the LED spot and the LED light, so that the VR handle can perform nonlinear optimization on the 6Dof initial pose of the VR handle in the world coordinate system at the current exposure moment in combination with the motion data collected by the built-in IMU to obtain the 6Dof target pose.

[0166] It should be noted that the frequency of sending data packets is not strictly fixed, that is, the time interval between the data packets corresponding to each two frames of the handle image is not fixed. Based on the computing power of the VR device, the frequency of sending data packets is close to the exposure frequency. The contents of the Part1-Part13 fields in the message sending the data packet are shown in Table 6.

[0167] Table 6. Contents of the Part1-Part13 fields in the message of the data packet sent by the VR device

[0168]

[0169] After receiving this message, the VR handle immediately replies with a message (indicated by a solid line) to notify the VR device that it has received the data packet. In the reply message of the VR handle to the message that the VR device sends the data packet, the contents of the Part1-Part9 fields are shown in Table 7.

[0170] Table 7. Contents of the Part1-Part9 fields in the reply message corresponding to the rendering command

[0171]

[0172]

[0173] In the method for positioning the VR handle in collaboration with the VR device, in order to avoid the influence of electromagnetic interference, signal occupancy, frequency band occupancy, etc., and to ensure the accuracy and robustness of the positioning results, during the communication process between the VR device and the VR handle, it is necessary to ensure that the VR handle's reply message to the exposure command issued by the VR device and the message of the VR device sending a data packet to the VR handle can be effectively transmitted. In the embodiment of the present application, the VR handle and the VR headset only need to transmit the data of the exposure moment and the rendering moment at a low frequency. When the use environment of the VR all-in-one machine affects the transmission of these two messages, data retransmission can be guaranteed within a limited time, thereby ensuring the positioning accuracy and robustness of the VR handle. Specifically, if Fig.11As shown, for the reply message of the exposure command, when the VR handle does not receive the confirmation message represented by the single-point dashed line, it will repeatedly send a certain amount of reply messages; for the message of the VR device sending a data packet, before the next 6Dof initial posture is calculated, a certain amount of messages represented by long and short dashed lines will be repeatedly sent.

[0174] It should be noted that when there is no conflict in the contents of the Part1-Part12 fields of the instructions and messages sent by the VR device to the VR controller, multiple types of instructions and messages can be combined into one message and sent. When there is a conflict in the contents of the Part1-Part12 fields of the instructions and messages sent by the VR device to the VR controller, they can be sent sequentially according to the exposure rendering order.

[0175] like Fig.12 As shown, in the VR handle positioning algorithm provided in the embodiment of the present application, the specific interaction process between the VR device and the VR handle mainly includes the following steps:

[0176] S1201: The VR controller calculates its own 3D pose in real time based on the motion data continuously collected by the built-in IMU.

[0177] S1202: The VR device sends an exposure instruction to the VR handle according to the camera exposure frequency. The exposure instruction includes the current exposure time of the camera and is used to instruct the VR handle to light up the LED light ring at the current exposure time.

[0178] S1203-S1204: The VR handle lights up the LED light ring according to the exposure instruction, and responds to the exposure instruction by repeatedly sending a first message including the 3Dof posture and 6Dof predicted posture of the VR handle at the current exposure moment to the VR device.

[0179] S1205: The VR device collects an image of the handle when the LED light ring is on.

[0180] S1206: After receiving the first message, the VR device returns a confirmation message to the VR handle to notify the VR handle that it has received the posture data of the current exposure moment sent by the VR handle.

[0181] S1207: The VR device performs light spot detection on the handle image collected at the current exposure moment to obtain the 2D coordinates of each light spot.

[0182] S1208: The VR device determines the 2D-3D mapping relationship between the LED light spot and the LED light according to the 3Dof posture or 6Dof predicted posture of the VR handle at the current exposure moment.

[0183] S1209: The VR device determines the 6Dof initial position of the VR handle in the world coordinate system according to the 2D-3D mapping relationship between the LED light spots and the LED lights, and the 2D coordinates of each light spot.

[0184] S1210: The VR device packages the 2D-3D mapping relationship between the LED light spots and the LED lights, the 2D coordinates of each light spot, and the 6DoF initial posture of the VR handle into a data packet, and sends it to the VR handle through a message.

[0185] S1211: The VR controller performs nonlinear optimization on the received 6DoF initial pose according to the motion data collected by the built-in IMU before receiving the data packet to obtain the 6DoF target pose.

[0186] S1212: The VR device sends a rendering instruction to the VR device according to the rendering frequency of the display, and the rendering instruction carries the current rendering time.

[0187] S1213: The VR controller responds to the rendering instruction, and sends its 6Dof predicted pose in the world coordinate system at the current rendering moment indicated by the rendering instruction to the VR device through a second message, wherein the 6Dof predicted pose is predicted by the VR controller based on the 6Dof target pose optimized at the most recent exposure moment.

[0188] S1214: The VR device renders and displays the VR handle in the current virtual screen according to the 6Dof predicted position of the VR handle in the world coordinate system at the current rendering moment.

[0189] S1215: The VR controller predicts the 6DoF target pose according to the optimized 6DoF target pose, and obtains the 6DoF predicted pose of the VR controller in the world coordinate system when the next frame of the controller image is exposed.

[0190] S1216: Before calculating the next 6Dof initial posture, the VR handle repeatedly sends the 6Dof predicted posture when the next frame of the handle image is exposed to the VR device, so that the VR device performs LED spot detection according to the 6Dof predicted posture.

[0191] S1217: The VR device sends a rendering instruction to the VR device according to the rendering frequency of the display, and the rendering instruction carries the next rendering time.

[0192] S1218: The VR controller responds to the rendering instruction and sends its 6Dof predicted pose in the world coordinate system at the next rendering moment indicated by the rendering instruction to the VR device through a second message. The 6Dof predicted pose is predicted by the VR controller based on the 6Dof target pose optimized at the most recent exposure moment.

[0193] S1219: The VR device renders and displays the VR handle in the next virtual screen according to the 6Dof predicted position of the VR handle in the world coordinate system at the next rendering moment.

[0194] The embodiment of the present application is based on near-infrared light spot projection positioning technology, nonlinear optimization technology and wireless device time synchronization technology, and aims at the use scenario of VR all-in-one machine, and redesigns the system architecture for calculating the 6DOf posture of VR handle. The VR handle and VR device jointly realize the 6DOf positioning of the VR handle, wherein the 3Dof posture calculation, nonlinear 6Dof posture optimization calculation and 6Dof posture prediction calculation process are deployed on the VR handle end, and the exposure imaging, LED spot detection, establishment of 2D-3D mapping relationship between light spot and LED light, and 6Dof initial posture calculation process of the VR handle are deployed on the VR device end. In this way, VR The handle does not need to transmit the motion data collected in real time by the IMU built into the VR handle to the VR device at high frequency. It only needs to transmit the pose calculation results (3Dof pose and 6DOf predicted pose) at the required time (including exposure time and rendering time) to the VR device at low frequency. When there are electromagnetic interference, channel occupancy, frequency band occupancy and other problems in the use environment of the VR all-in-one machine, the VR handle can ensure that data is not lost by retransmitting data, thereby preventing the impact of data loss on the 6Dof pose calculation of the VR handle, improving positioning accuracy and robustness, while reducing the pressure on transmission bandwidth, improving positioning efficiency, and effectively increasing the breadth of VR all-in-one usage scenarios.

[0195] Based on the same technical concept, an embodiment of the present application provides a VR handle, which can jointly implement the above-mentioned VR handle positioning method with a VR device synchronized with the timeline, and can achieve the same technical effect.

[0196] See also Fig.13 The VR handle includes a near-infrared LED light ring 1301, an IMU 1302, a memory 1303, a processor 1304, and a communication interface 1305. The near-infrared LED light ring 1301, the IMU 1302, the memory 1303, the processor 1304, and the communication interface 1305 are connected via a bus 1306;

[0197] The memory 1303 stores a computer program, and the processor 1304 performs the following operations according to the computer program:

[0198] Calculate the 3Dof posture of the device in real time based on the motion data continuously collected by the IMU 1302;

[0199] Receive the exposure instruction sent by the VR device according to the camera exposure frequency through the communication interface 1305, and light up the near-infrared LED light ring 1301 according to the current exposure time indicated by the exposure instruction;

[0200] Repeating sending a first message in response to the exposure instruction to the VR device through the communication interface 1305 until receiving feedback from the VR device on the first message; wherein the first message includes a 3Dof pose of the VR handle at the current exposure moment and a 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment, and the 3Dof pose and the 6Dof predicted pose are used to determine a 2D-3D mapping relationship between an LED light spot and an LED light;

[0201] Receiving, through the communication interface 1305, a data packet repeatedly sent by the VR device before calculating the next 6DOf real pose, the data packet including the 2D-3D mapping relationship between the LED light spot and the LED light, the 2D coordinates of each LED light spot in the handle image collected at the current exposure moment, and the 6DOf initial pose of the VR handle in the world coordinate system at the current exposure moment;

[0202] Through the communication interface 1305, according to the received data packet and the motion data collected by the IMU before the data packet is received, the 6Dof initial posture of the VR handle in the world coordinate system is nonlinearly optimized to obtain the 6Dof target posture;

[0203] Determine the 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment according to the 6Dof target pose;

[0204] Through the communication interface 1305, a rendering instruction sent by the VR device is received, and a second message in response to the rendering instruction is sent to the VR device, wherein the second message carries the 6Dof predicted posture, so that the VR device renders and displays the VR handle in the current screen according to the 6Dof predicted posture.

[0205] Optionally, after obtaining the 6Dof target pose, the processor 1304 further executes:

[0206] Determine the 6Dof predicted pose of the VR controller in the world coordinate system at the next exposure moment according to the 6Dof target pose of the VR controller in the world coordinate system at the current exposure moment;

[0207] Before the next exposure moment arrives, the 6Dof predicted position of the VR handle at the next exposure moment is repeatedly sent to the VR device, so that the VR device determines the area of ​​the VR handle in the handle image at the next exposure moment.

[0208] After the optional processor 1304 determines the 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment, it further executes:

[0209] Determine the 6Dof predicted pose of the VR controller in the world coordinate system at the next rendering moment according to the 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment;

[0210] Before the next rendering moment arrives, the 6Dof predicted posture of the VR handle at the next rendering moment is repeatedly sent to the VR device to achieve posture smoothing.

[0211] Optionally, the message sent by the VR handle to the VR device includes at least the length of the message, the timestamp of the last received LED light ring data, the timestamp of the 3Dof pose, the 3Dof pose, the timestamp of the current exposure moment, the 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment, the timestamp of the current rendering moment, the 6Dof pose of the VR handle in the world coordinate system at the current rendering moment and a check code field.

[0212] It should be noted that Fig.13 This is just an example, and provides the necessary hardware for a VR handle to execute the steps of a VR handle positioning method provided in an embodiment of the present application. If not shown, the VR handle may also include trigger keys, joysticks, buttons, and other hardware of a conventional VR handle.

[0213] Based on the same technical concept, an embodiment of the present application provides a VR device, which can implement the above-mentioned VR handle positioning method together with a VR handle synchronized with the timeline, and can achieve the same technical effect.

[0214] See also Fig.14 , the VR device includes a plurality of near-infrared cameras 1401, a display 1402, a memory 1403, a processor 1404 and a communication interface 1405, and the plurality of near-infrared cameras 1401, the display 1402, the memory 1403, the processor 1404 and the communication interface 1405 are connected via a bus 1406;

[0215] The memory 1403 stores a computer program, and the processor 1404 performs the following operations according to the computer program:

[0216] Through the communication interface 1405, an exposure instruction is sent to the VR handle according to the exposure frequency of the near-infrared camera 1401, an image of the handle when the LED light ring is on at the current exposure moment is obtained, and LED spot detection is performed on the handle image to obtain the 2D coordinates of each LED spot;

[0217] Receiving, through the communication interface 1405, a first message sent by the VR handle in response to the exposure instruction, if the handle image is a first-frame exposure image, determining a 2D-3D mapping relationship between the LED light spot and the LED light according to the 3Dof pose of the VR handle at the current exposure moment contained in the first message; otherwise, determining a 2D-3D mapping relationship between the LED light spot and the LED light according to the 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment contained in the first message;

[0218] Determine the 6Dof initial position of the VR handle in the world coordinate system at the current exposure moment according to the 2D-3D mapping relationship between the LED light spots and the LED lights and the 2D coordinates of each LED light spot;

[0219] Before the next 6Dof initial posture calculation, the 2D-3D mapping relationship between the LED light spot and the LED light, the 2D coordinates of each LED light spot and the 6Dof initial posture of the VR handle are packaged into a data packet and repeatedly sent to the VR handle through the communication interface 1405;

[0220] Sending a rendering instruction to the VR controller according to the rendering frequency of the display 1402 through the communication interface 1405;

[0221] Through the communication interface 1405, a second message in which the VR handle replies to the exposure instruction is received, and the VR handle is rendered in the current screen according to the 6Dof predicted posture of the VR handle at the current rendering moment carried in the second message, and displayed through the display.

[0222] Optionally, the processor 1404 determines the 2D-3D mapping relationship between the LED light spot and the LED light according to the 3Dof posture of the VR handle at the current exposure moment included in the first message, and the specific operation is:

[0223] Determine a 2D-3D mapping relationship between multiple groups of LED light spots and LED lights according to the 2D coordinates of each LED light spot and the 3D structure of the LED light ring;

[0224] For each set of 2D-3D mapping relationships, perform the following operations:

[0225] Obtain the 3D coordinates of the LED lights corresponding to each of the LED light spots, and determine the 6Dof pose of the VR handle in the camera coordinate system by combining the 2D coordinates of each of the LED light spots and pre-calibrated camera internal parameters;

[0226] According to the 6DOf pose of the VR device in the camera coordinate system, the pre-calibrated external parameters between the camera and the built-in IMU, the 6DOf pose of the VR device in the world coordinate system at the current exposure moment is obtained;

[0227] Determine the 6Dof initial pose of the VR handle in the world coordinate system at the current exposure moment according to the 6Dof pose of the VR device in the world coordinate system and the 6Dof pose of the VR handle in the camera coordinate system at the current exposure moment;

[0228] The 6Dof initial poses corresponding to the multiple groups of 2D-3D mapping relationships are verified using the 3Dof pose of the VR handle, and a group of 2D-3D mapping relationships is selected.

[0229] Optionally, the processor 1404 determines the 2D-3D mapping relationship between the LED light spot and the LED light according to the 6Dof predicted posture of the VR handle in the world coordinate system at the current exposure moment contained in the first message, and the specific operation is:

[0230] According to the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure moment and the pre-calibrated camera internal parameters, the LED light ring is projected into the controller image at the current exposure moment;

[0231] According to the projection coordinates of each LED light spot in the handle image and the 2D coordinates of each LED light spot in the handle image, a 2D-3D mapping relationship between a group of LED light spots and LED lamps is directly determined.

[0232] Optionally, the instruction sent by the VR device to the VR handle includes at least the length of this message, a flag of whether the LED light ring is on, a flag of whether the 3Dof pose of the VR handle is received within the LED light ring lighting cycle, the timestamp of the current exposure moment, the timestamp of the current rendering moment, the timestamp of the 6Dof initial pose of the VR handle calculated at the most recent exposure moment, the 6Dof target pose of the VR handle at the last exposure moment, the first serial number of the camera that observes the most LED spots, the number of LED spots observed by the camera with the first serial number, the second serial number of the camera that observes the second most LED spots, the number of LED spots observed by the camera with the second serial number, and the 2D coordinates and check code fields of each LED spot observed by the camera with the first serial number and the camera with the second serial number described in sequence.

[0233] It should be noted that Fig.14 This is just an example, and provides the necessary hardware for a VR device to execute the steps of a VR handle positioning method provided in an embodiment of the present application. If not shown, the VR device may also include conventional hardware such as an IMU, a speaker, a pickup, an operation key, and a power supply.

[0234] It should be noted that Fig.13 and Fig.14 The memory in the term may be a volatile memory, such as a random access memory; the memory may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid state drive; or the memory may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories; the processor may include one or more central processing units or a digital processing unit, etc.

[0235] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete a VR handle positioning method in the aforementioned embodiment.

[0236] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a VR handle positioning method in the aforementioned embodiment.

[0237] Those skilled in the art should understand that various changes and modifications can be made to the present application without departing from the idea and scope of the present application, and these changes and modifications belong to the scope of the claims of the present application and their equivalent technologies.

Claims

1. A VR handle positioning method, characterized in that: Applied to a VR handle, the VR handle is synchronized with the time axis of the VR device, and the method includes: Calculate the 3D pose of the VR controller in real time based on the motion data continuously collected by the built-in IMU; Receive an exposure instruction sent by the VR device according to the camera exposure frequency, and light up the LED light ring according to the current exposure time indicated by the exposure instruction; Repeating sending a first message in response to the exposure instruction to the VR device until receiving feedback from the VR device on the first message; wherein the first message includes a 3Dof pose of the VR handle at the current exposure moment and a 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment, and the 3Dof pose and the 6Dof predicted pose are used to determine a 2D-3D mapping relationship between an LED light spot and an LED light; Receive a data packet repeatedly sent by the VR device before calculating the next 6DOf real pose of the VR handle in the world coordinate system, the data packet including a 2D-3D mapping relationship between LED light spots and LED lights, a 2D coordinate of each LED light spot in a handle image collected at the current exposure time, and a 6DOf initial pose of the VR handle in the world coordinate system at the current exposure time; According to the received data packet and the motion data collected by the built-in IMU before the data packet is received, nonlinearly optimize the 6Dof initial posture of the VR handle in the world coordinate system to obtain the 6Dof target posture; Determine the 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment according to the 6Dof target pose; Receive a rendering instruction sent by the VR device, and send a second message in response to the rendering instruction to the VR device, wherein the second message carries the 6Dof predicted posture, so that the VR device renders and displays the VR handle in the current screen according to the 6Dof predicted posture.

2. The method according to claim 1, characterized in that After obtaining the 6Dof target pose, the method further includes: Determine the 6Dof predicted pose of the VR controller in the world coordinate system at the next exposure moment according to the 6Dof target pose of the VR controller in the world coordinate system at the current exposure moment; Before the next exposure moment arrives, the 6Dof predicted position of the VR handle at the next exposure moment is repeatedly sent to the VR device, so that the VR device determines the area of ​​the VR handle in the handle image at the next exposure moment.

3. The method according to claim 1, characterized in that After determining the 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment, the method further includes: Determine the 6Dof predicted position of the VR controller in the world coordinate system at the next rendering moment according to the 6Dof predicted position of the VR controller in the world coordinate system at the current rendering moment; Before the next rendering moment arrives, the 6Dof predicted posture of the VR handle at the next rendering moment is repeatedly sent to the VR device to achieve posture smoothing.

4. The method according to any one of claims 1 to 3, characterized in that The message sent by the VR handle to the VR device includes at least the length of the message, the timestamp of the last received LED light ring data, the timestamp of the 3Dof pose, the 3Dof pose, the timestamp of the current exposure moment, the 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment, the timestamp of the current rendering moment, the 6Dof pose of the VR handle in the world coordinate system at the current rendering moment, and a check code field.

5. A VR handle positioning method, characterized in that: Applied to a VR device, the VR device is synchronized with the time axis of a VR controller, and the method includes: Sending an exposure command to the VR handle according to the camera exposure frequency, obtaining a handle image captured by the camera when the LED light ring is on at the current exposure moment, and performing LED spot detection on the handle image to obtain the 2D coordinates of each LED spot; Receive a first message sent by the VR handle in response to the exposure instruction, and if the handle image is a first frame exposure image, determine the 2D-3D mapping relationship between the LED light spot and the LED light according to the 3Dof pose of the VR handle at the current exposure moment contained in the first message; otherwise, determine the 2D-3D mapping relationship between the LED light spot and the LED light according to the 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment contained in the first message; Determine the 6Dof initial position of the VR handle in the world coordinate system at the current exposure moment according to the 2D-3D mapping relationship between the LED light spots and the LED lights and the 2D coordinates of each LED light spot; Encapsulate the 2D-3D mapping relationship between the LED light spots and the LED lights, the 2D coordinates of each LED light spot, and the 6Dof initial pose of the VR handle into a data packet, and repeatedly send it to the VR handle before calculating the next 6Dof initial pose of the VR handle in the world coordinate system; Sending a rendering instruction to the VR controller according to the rendering frequency to obtain a 6Dof predicted position of the VR controller in the world coordinate system at the current rendering moment indicated by the rendering instruction; A second message is received in which the VR controller replies to the exposure instruction, and the VR controller is rendered and displayed in the current screen according to the 6Dof predicted posture of the VR controller at the current rendering moment carried in the second message.

6. The method according to claim 5, characterized in that The determining, according to the 3Dof posture of the VR handle at the current exposure moment contained in the first message, a 2D-3D mapping relationship between the LED light spot and the LED light includes: Determine a 2D-3D mapping relationship between multiple groups of LED light spots and LED lights according to the 2D coordinates of each LED light spot and the 3D structure of the LED light ring; For each set of 2D-3D mapping relationships, perform the following operations: Obtain the 3D coordinates of the LED lights corresponding to each of the LED light spots, and determine the 6Dof pose of the VR handle in the camera coordinate system by combining the 2D coordinates of each of the LED light spots and pre-calibrated camera internal parameters; According to the 6DOf pose of the VR device in the camera coordinate system, the pre-calibrated external parameters between the camera and the built-in IMU, the 6DOf pose of the VR device in the world coordinate system at the current exposure moment is obtained; Determine the 6Dof initial pose of the VR handle in the world coordinate system at the current exposure moment according to the 6Dof pose of the VR device in the world coordinate system and the 6Dof pose of the VR handle in the camera coordinate system at the current exposure moment; The 6Dof initial poses corresponding to the multiple groups of 2D-3D mapping relationships are verified using the 3Dof pose of the VR handle, and a group of 2D-3D mapping relationships is selected.

7. The method according to claim 5, characterized in that The determining of the 2D-3D mapping relationship between the LED light spot and the LED light according to the 6Dof predicted position of the VR handle in the world coordinate system at the current exposure moment contained in the first message includes: According to the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure moment and the pre-calibrated camera internal parameters, the LED light ring is projected into the controller image at the current exposure moment; According to the projection coordinates of each LED light spot in the handle image and the 2D coordinates of each LED light spot in the handle image, a 2D-3D mapping relationship between a group of LED light spots and LED lamps is directly determined.

8. The method according to any one of claims 5 to 7, characterized in that The instruction sent by the VR device to the VR handle at least includes the length of this message, a flag indicating whether the LED light ring is on, a flag indicating whether the 3Dof pose of the VR handle is received within the LED light ring lighting cycle, a timestamp of the current exposure moment, a timestamp of the current rendering moment, a timestamp of the 6Dof initial pose of the VR handle calculated at the most recent exposure moment, the 6Dof target pose of the VR handle at the last exposure moment, the first serial number of the camera that observes the most LED spots, the number of LED spots observed by the camera with the first serial number, the second serial number of the camera that observes the second most LED spots, the number of LED spots observed by the camera with the second serial number, and the 2D coordinates and check code fields of each LED spot observed by the camera with the first serial number and the camera with the second serial number described in sequence.

9. A VR handle, characterized in that: The VR device is synchronized with the time axis of a VR handle, wherein the VR handle includes a near-infrared LED light ring, an IMU, a memory, a processor, and a communication interface, and the near-infrared LED light ring, the IMU, the memory, the communication interface, and the processor are connected via a bus; The memory stores a computer program, and the processing performs the following operations according to the computer program: Calculate the 3D pose of the device in real time based on the motion data continuously collected by the IMU; Receiving, via the communication interface, an exposure instruction sent by the VR device according to the camera exposure frequency, and lighting up the near-infrared LED light ring according to the current exposure time indicated by the exposure instruction; Repeating, through the communication interface, sending a first message in response to the exposure instruction to the VR device until receiving feedback from the VR device on the first message; wherein the first message includes a 3Dof pose of the VR handle at the current exposure moment and a 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment, and the 3Dof pose and the 6Dof predicted pose are used to determine a 2D-3D mapping relationship between an LED light spot and an LED light; Receiving, through the communication interface, a data packet repeatedly sent by the VR device before calculating the next 6DOf real pose, the data packet including a 2D-3D mapping relationship between LED light spots and LED lights, a 2D coordinate of each LED light spot in a handle image collected at the current exposure moment, and a 6DOf initial pose of the VR handle in the world coordinate system at the current exposure moment; Through the communication interface, according to the received data packet and the motion data collected by the IMU before the data packet is received, nonlinearly optimize the 6Dof initial posture of the VR handle in the world coordinate system to obtain the 6Dof target posture; Determine the 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment according to the 6Dof target pose; Through the communication interface, a rendering instruction sent by the VR device is received, and a second message in response to the rendering instruction is sent to the VR device, wherein the second message carries the 6Dof predicted posture, so that the VR device renders and displays the VR handle in the current screen according to the 6Dof predicted posture.

10. A VR device, characterized in that: The time axis of the VR device and the VR handle are synchronized, wherein the VR device includes a plurality of near-infrared cameras, a display, a memory, a processor, and a communication interface; The memory stores a computer program, and the processor performs the following operations according to the computer program: Through the communication interface, an exposure instruction is sent to the VR handle according to the exposure frequency of the near-infrared camera, an image of the handle when the LED light ring is on at the current exposure moment is obtained, and LED spot detection is performed on the handle image to obtain the 2D coordinates of each LED spot; Receiving, through the communication interface, a first message sent by the VR handle in response to the exposure instruction, if the handle image is a first-frame exposure image, determining a 2D-3D mapping relationship between the LED light spot and the LED light according to the 3Dof pose of the VR handle at the current exposure moment contained in the first message; otherwise, determining a 2D-3D mapping relationship between the LED light spot and the LED light according to the 6Dof predicted pose of the VR handle in the world coordinate system at the current exposure moment contained in the first message; Determine the 6Dof initial position of the VR handle in the world coordinate system at the current exposure moment according to the 2D-3D mapping relationship between the LED light spots and the LED lights and the 2D coordinates of each LED light spot; Before the next 6Dof initial posture calculation, the 2D-3D mapping relationship between the LED light spot and the LED light, the 2D coordinates of each LED light spot and the 6Dof initial posture of the VR handle are packaged into a data packet and repeatedly sent to the VR handle through the communication interface; Sending a rendering instruction to the VR controller through the communication interface according to the rendering frequency of the display; Through the communication interface, a second message in which the VR handle replies to the exposure instruction is received, and the VR handle is rendered in the current screen according to the 6Dof predicted posture of the VR handle at the current rendering moment carried in the second message, and displayed through the display.

Images