VR handle positioning method and device

Abstract

The application relates to the VR technical field and provides a VR handle positioning method and equipment. The VR handle performs self 3Dof pose calculation, nonlinear 6Dof pose optimization calculation and 6Dof pose prediction calculation, the VR equipment performs near-infrared LED lamp ring exposure imaging, LED light spot detection, construction of a 2D-3D mapping relationship between the LED light spot and the LED lamp, 6Dof initial pose calculation of the VR handle. In this way, the VR handle only needs to transmit the pose calculation result of the VR equipment at the required moment to the VR equipment at a low frequency, the transmission bandwidth pressure is reduced, and when there are problems such as electromagnetic interference, channel occupation and frequency band occupation, the VR handle can ensure that data is not lost through data retransmission, thereby improving the positioning accuracy and robustness of the VR handle. Meanwhile, through the cooperative calculation of the two devices, the positioning efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of virtual reality (VR) technology, and provides a VR controller positioning method and device. Background Technology

[0002] Generally, a VR all-in-one device consists of two parts: the VR device and the VR controller. Controller positioning is a key function of VR all-in-one devices, allowing users to subjectively experience the three-dimensional interaction between their hands and the virtual scene.

[0003] Currently, computer vision and inertial navigation technologies are mainly used for controller positioning. By deploying controller positioning algorithms on the VR device, the VR controllers transmit high-frequency data (e.g., 500Hz) collected by their inertial measurement units (IMUs) to the VR device via wireless communication. The VR device's system-on-chip (SoC) uses the controller positioning algorithm to calculate the controller's six degrees of freedom (DOF) based on the IMU data and the image of the LED ring on the VR controller captured by the camera. However, when the communication frequency band or channel between the VR device and the VR controller is occupied, or when electromagnetic interference exists in the environment, data frame loss may occur during positioning. Due to limitations in wireless communication bandwidth, the high-frequency IMU data from the controllers cannot be retransmitted on a large scale to ensure no frame loss, especially when data from two VR controllers needs to be transmitted simultaneously. This can lead to errors in pose calculation on the VR device side, resulting in inaccurate or failed positioning results. Summary of the Invention

[0004] This application provides a VR controller positioning method and device to improve the robustness of controller positioning.

[0005] On one hand, embodiments of this application provide a VR controller positioning algorithm applied to a VR controller, wherein the VR controller is synchronized with the timeline of a VR device, and the method includes:

[0006] The 3Dof pose of the VR controller is calculated in real time based on motion data continuously collected by the built-in IMU.

[0007] Receive the exposure command sent by the VR device according to the camera exposure frequency, and light up the LED ring according to the current exposure time indicated by the exposure command;

[0008] The system repeatedly sends a first message responding to the exposure command to the VR device until it receives feedback from the VR device on the first message; wherein, the first message includes the 3Dof pose of the VR controller at the current exposure time and the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time, and the 3Dof pose and the 6Dof predicted pose are used to determine the 2D-3D mapping relationship between the LED spot and the LED light.

[0009] The VR device receives data packets repeatedly sent before calculating the next 6DOf true pose of the VR controller in the world coordinate system. The data packets contain the 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot in the controller image acquired at the current exposure time, and the initial 6DOf pose of the VR controller in the world coordinate system at the current exposure time.

[0010] Based on the received data packet and the motion data collected by the built-in IMU before the data packet was received, the initial 6DoF pose of the VR controller in the world coordinate system is nonlinearly optimized to obtain the target 6DoF pose.

[0011] Based on the target 6DoF pose, determine the predicted 6DoF pose of the VR controller in the world coordinate system at the current rendering moment;

[0012] The system receives a rendering instruction sent by the VR device and sends a second message responding to the rendering instruction to the VR device. The second message carries the 6DoF predicted pose, so that the VR device renders and displays the VR controller in the current screen according to the 6DoF predicted pose.

[0013] On the other hand, embodiments of this application provide a VR controller positioning method applied to a VR device, wherein the VR device and the VR controller's time axis are synchronized, and the method includes:

[0014] According to the camera's exposure frequency, an exposure command is sent to the VR controller to obtain the controller image captured by the camera when the LED ring is lit at the current exposure moment, and the controller image is subjected to LED spot detection to obtain the 2D coordinates of each LED spot;

[0015] The system receives a first message sent by the VR controller in response to the exposure command. If the controller image is the first frame exposure image, the system determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 3Dof pose of the VR controller at the current exposure time contained in the first message. Otherwise, the system determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time contained in the first message.

[0016] Based on the 2D-3D mapping relationship between the LED spot and the LED light and the 2D coordinates of each LED spot, the initial 6Dof pose of the VR controller in the world coordinate system at the current exposure time is determined;

[0017] The 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot, and the 6DoF initial pose of the VR controller are encapsulated into a data packet, and repeatedly sent to the VR controller before calculating the next 6DoF initial pose of the VR controller in the world coordinate system.

[0018] According to the rendering frequency, a rendering command is sent to the VR controller to obtain the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering time indicated by the rendering command.

[0019] The system receives a second message from the VR controller in response to the exposure command, and renders and displays the VR controller in the current screen based on the 6DoF predicted pose of the VR controller at the current rendering time carried in the second message.

[0020] On the other hand, this application provides a VR controller that is synchronized with the timeline of a VR device. The VR controller includes a near-infrared LED ring, an IMU, a memory, a processor, and a communication interface. The near-infrared LED ring, the IMU, the memory, and the processor are connected via a bus.

[0021] The memory stores a computer program, and the processing performs the following operations according to the computer program:

[0022] Based on the motion data continuously collected by the IMU, the 3Dof pose of itself is calculated in real time.

[0023] The system receives exposure commands from the VR device according to the camera's exposure frequency via the communication interface, and illuminates the near-infrared LED ring according to the current exposure time indicated by the exposure commands.

[0024] Through the communication interface, the first message responding to the exposure command is repeatedly sent to the VR device until the feedback of the VR device to the first message is received; wherein, the first message includes the 3Dof pose of the VR controller at the current exposure time and the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time, and the 3Dof pose and the 6Dof predicted pose are used to determine the 2D-3D mapping relationship between the LED spot and the LED light.

[0025] Through the communication interface, the VR device receives data packets repeatedly sent before calculating the next 6DOf real pose. The data packets include the 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot in the controller image acquired at the current exposure time, and the initial 6DOf pose of the VR controller in the world coordinate system at the current exposure time.

[0026] Through the communication interface, based on the received data packet and the motion data collected by the IMU before the data packet was received, the initial 6DoF pose of the VR controller in the world coordinate system is nonlinearly optimized to obtain the target 6DoF pose.

[0027] Based on the target 6DoF pose, determine the predicted 6DoF pose of the VR controller in the world coordinate system at the current rendering moment;

[0028] The system receives rendering instructions from the VR device via the communication interface and sends a second message in response to the rendering instructions to the VR device. The second message carries the 6DoF predicted pose, so that the VR device renders and displays the VR controller in the current screen according to the 6DoF predicted pose.

[0029] On the other hand, this application embodiment provides a VR device with synchronized timelines between the VR device and the VR controller. The VR device includes multiple near-infrared cameras, a display, a memory, a processor, and a communication interface.

[0030] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0031] Through the communication interface, an exposure command is sent to the VR controller according to the exposure frequency of the near-infrared camera, the controller image when the LED ring is lit at the current exposure moment is obtained, and the LED spot detection is performed on the controller image to obtain the 2D coordinates of each LED spot;

[0032] Through the communication interface, the system receives a first message sent by the VR controller in response to the exposure command. If the controller image is the first frame exposure image, the system determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 3Dof pose of the VR controller at the current exposure time contained in the first message. Otherwise, the system determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time contained in the first message.

[0033] Based on the 2D-3D mapping relationship between the LED spot and the LED light and the 2D coordinates of each LED spot, the initial 6Dof pose of the VR controller in the world coordinate system at the current exposure time is determined;

[0034] Before the next 6DoF initial pose calculation, the 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot, and the 6DoF initial pose of the VR controller are encapsulated into a data packet and repeatedly sent to the VR controller through the communication interface.

[0035] Through the communication interface, rendering commands are sent to the VR controller according to the rendering frequency of the display;

[0036] The system receives a second message from the VR controller in response to the exposure command via the communication interface. Based on the 6DoF predicted pose of the VR controller at the current rendering time carried in the second message, the system renders the VR controller in the current screen and displays it on the monitor.

[0037] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the steps of a VR controller positioning method provided in embodiments of this application.

[0038] In the VR controller positioning method and device provided in this application embodiment, the VR controller and VR device are synchronized on the time axis. Through the interaction between the VR device and the VR controller, the 6DoF pose of the VR controller is jointly located. During the positioning process, the VR controller calculates its own 3DoF pose in real time based on the motion data continuously collected by the built-in IMU. After receiving the exposure command sent by the VR device, the VR controller lights up the LED ring. At this time, the camera of the VR device captures the image of the controller when the LED ring is lit and determines the 2D coordinates of each LED spot. For the exposure command, the VR controller repeatedly sends the 3DoF pose of the VR controller at the current exposure time or the predicted 6DoF pose of the VR controller in the world coordinate system in the first message to the VR device to prevent data loss. The VR device determines the 2D-3D mapping relationship between LED spots and LED lights based on the first message, and obtains the 3D coordinates of the LED lights corresponding to each LED spot based on the 2D-3D mapping relationship. Combining this with the 2D coordinates of the corresponding LED spots, it determines the initial 6DoF pose of the VR controller in the world coordinate system at the current exposure moment. The 2D-3D mapping relationship, the 2D coordinates of each LED spot, and the initial 6DoF pose of the VR controller are packaged and repeatedly sent to the VR controller to prevent data loss. The VR controller, combined with motion data acquired by the IMU before receiving the data packet, performs nonlinear optimization on its initial 6DoF pose. Upon receiving the rendering command from the VR device, it sends the predicted 6DoF pose of the VR controller in the world coordinate system at the current rendering moment, determined based on the optimized 6DoF target pose, to the VR device via the second message. The VR device renders and displays the VR controller in the current screen based on the predicted 6DoF pose of the VR controller at the current rendering moment to reflect the user's hand interaction with the virtual scene in three dimensions. By having both the VR controller and the VR device jointly execute the controller positioning algorithm, the computational resource consumption on the VR device side is reduced. During the positioning process, the VR device only needs to send the current data to the VR device at a low frequency. This allows the impact of electromagnetic interference, channel occupancy, and frequency band occupancy on pose calculations to be avoided through data retransmission, thereby improving the positioning accuracy and robustness of the VR controller, while reducing the pressure on transmission bandwidth and improving positioning efficiency. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A schematic diagram of a VR all-in-one device provided in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0042] Figure 3 This is an overall architecture diagram of the VR controller positioning method provided in the embodiments of this application;

[0043] Figure 4 A flowchart illustrating the implementation of the VR controller positioning algorithm provided in this application embodiment;

[0044] Figure 5 The format for a single message sent from a VR controller to a VR device;

[0045] Figure 6 A flowchart illustrating the implementation process of the VR controller positioning algorithm on the VR device side.

[0046] Figure 7 This is a flowchart of a method for establishing a 2D-3D mapping relationship between an LED light spot and an LED lamp;

[0047] Figure 8 Flowchart of another method for establishing a 2D-3D mapping relationship between LED light spots and LED lights;

[0048] Figure 9 The format for a single message sent from a VR device to a VR controller;

[0049] Figure 10 A schematic diagram illustrating the communication mechanism between VR devices and VR controllers when unaffected by the usage environment;

[0050] Figure 11 This is a schematic diagram illustrating the communication mechanism between VR devices and VR controllers when affected by the usage environment.

[0051] Figure 12 A flowchart illustrating the interaction method between VR devices and VR controllers;

[0052] Figure 13 This is a structural diagram of a VR controller;

[0053] Figure 14 The structure of a VR device. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0055] like Figure 1 The diagram shown illustrates a VR all-in-one device provided in this application. A typical VR all-in-one device includes a VR headset and VR controllers. The VR headset contains multiple near-infrared cameras, each uniquely identified by a camera serial number (e.g., Arabic numerals 1-4). An IMU (Insulated Unit) is installed inside the VR headset to collect head motion data. Through pre-calibration, the intrinsic parameters of each camera and the extrinsic parameters between the camera and the VR headset's built-in IMU can be obtained. Each VR controller has a ring-shaped fixed housing made of a material with high infrared light transmittance. Several near-infrared LED light sources are encapsulated under the housing, forming an LED light ring with a fixed 3D structure. The 3D coordinates of each near-infrared LED on the LED light ring are known, and each near-infrared LED has a unique identifier. An IMU is also installed inside the VR controller to collect the controller's motion data.

[0056] Typically, VR all-in-one devices synchronize the timelines of the VR device and VR controllers before leaving the factory, ensuring that the timestamps of the SOC systems of the two devices are aligned at the same moment. After timeline synchronization, the VR controllers can clearly determine the exposure time when the VR device's camera captures the next frame of the controller image and the rendering time of the next frame of the controller image. Furthermore, the VR controllers have pre-calibrated camera parameters of the VR device and extrinsic parameters between the camera and the VR device's built-in IMU, so that the VR controllers can calculate reprojection errors when performing nonlinear optimization of 6DoF pose.

[0057] like Figure 2 The diagram shows an application scenario of a VR all-in-one device. Users control the virtual screen of the VR device by operating the VR controller with their hands. Therefore, controller positioning is a key function of the VR all-in-one device, and its positioning accuracy directly affects the accuracy of virtual screen control. Furthermore, through the rendering and display of the controller, users can subjectively experience the three-dimensional interaction process between their hands and the virtual scene.

[0058] Currently, controller positioning algorithms are primarily deployed on the VR device itself. After the positioning process starts, the VR device uses a camera to image the LED ring on the VR controller and continuously receives motion data collected at high frequencies from the IMU built into the VR controller to calculate the controller's 6DoF pose. Since VR all-in-one devices are frequently used in environments containing Bluetooth, WiFi, and other wireless devices (such as shopping malls, conference venues, and multi-device interaction scenarios), the high-frequency transmission of motion data collected by the VR controller's built-in IMU to the VR device via 2.4GHz wireless communication technology may encounter issues such as electromagnetic interference, frequency band occupancy, and channel occupancy. This can cause frame loss in the VR controller's IMU data, preventing the VR device from effectively tracking the controller and resulting in incorrect or failed VR controller positioning.

[0059] To address the impact of electromagnetic interference, frequency band occupancy, and channel occupancy on VR controller positioning during VR all-in-one device use, this application provides a VR controller positioning algorithm. The algorithm involves the VR controller performing 3Dof pose calculation, nonlinear 6Dof pose optimization calculation, and 6Dof pose prediction calculation based on motion data collected by its built-in IMU. Meanwhile, the VR device performs controller image acquisition, spot detection, calculation of the 2D-3D mapping relationship between the LED spot and the LED light, and initial 6Dof pose calculation. This collaborative positioning algorithm, based on two distributed platforms—the VR controller and the VR device—reduces the consumption of computing resources on the VR device side. Furthermore, during positioning, the VR controller does not need to transmit the motion data collected by its built-in IMU to the VR device at high frequency via wireless communication technology; it only needs to transmit the pose calculation results to the VR device at low frequency and repeatedly. This prevents data loss from affecting the VR controller's 6Dof pose calculation, improves positioning accuracy and robustness, reduces transmission bandwidth pressure, and increases positioning efficiency.

[0060] See Figure 3This is an overall architecture diagram of the VR controller positioning method provided in this application embodiment. The VR device has a built-in IMU and a near-infrared camera, while the VR controller has a built-in IMU and a near-infrared LED ring. The timelines of the VR device and the VR controller are synchronized. The near-infrared camera on the VR device is used to expose and image the near-infrared LED ring on the VR controller. The IMU includes an accelerometer and an angular velocity meter, capable of measuring 3-axis acceleration and 3-axis angular velocity. The VR device's SLAM system can output its own 6DoF pose based on the 3-axis acceleration and 3-axis angular velocity of head movement collected by the built-in IMU and the image collected by the near-infrared camera, combined with the calibration parameters of the IMU and the camera. The VR controller calculates its own 3DoF pose in real time based on the 3-axis acceleration and 3-axis angular velocity of hand movement collected by the built-in IMU, and transmits the 3DoF pose required for the near-infrared camera exposure time to the VR device. When the LED ring is lit, the VR device exposes and images the VR controller. Based on the 6DOf predicted pose of the VR controller at the current exposure moment, the device determines the area of ​​the VR controller in the exposed image, thereby detecting LED spots within that area and obtaining the 2D coordinates of each LED spot. The VR device establishes a mapping relationship between the 2D LED spots and the 3Dof pose of the VR controller, or the 6DOf predicted pose of the VR controller, based on the 2D coordinates of each LED spot and the 3D LED light. Based on this mapping relationship and the 2D coordinates of each spot, the device uses the PnP (n≥3) algorithm to calculate the initial 6DOf pose of the VR controller. The VR device packages and sends the initial 6DOf pose of the VR controller, the 2D coordinates of each LED spot, and the mapping relationship to the VR controller. The VR controller, combined with motion data collected by the IMU, uses the Ceres math library to perform nonlinear optimization on the initial 6DOf pose to obtain the target 6DOf pose of the VR controller at the current exposure moment. Based on the target 6DOf pose, a prediction is made, and the prediction result is sent to the VR device via 2.4GHz wireless communication technology.

[0061] The following describes the specific process of the VR controller positioning algorithm from both the VR controller end and the VR device end.

[0062] like Figure 4 The diagram shown is a flowchart of the VR controller positioning algorithm implementation on the VR controller side. The process mainly includes the following steps:

[0063] S401: Calculates the 3Dof pose of the VR controller in real time based on motion data continuously collected by the built-in IMU.

[0064] In one example, motion data includes 3-axis acceleration and 3-axis angular velocity. Using algorithms such as complementary filtering or Kalman filtering, the 3DoF pose of the VR controller can be calculated in real time. The calculation result is a quaternion, and the 3D rotational attitude can be obtained by transforming the quaternion with the rotation matrix.

[0065] S402: Receives the exposure command sent by the VR device according to the camera's exposure frequency, and lights up the LED ring according to the current exposure time indicated by the exposure command.

[0066] To conserve energy, the LED ring on the VR controller flashes, requiring the near-infrared camera on the VR device to expose itself simultaneously with the LED ring illuminating. Therefore, the VR device sends an exposure command to the VR controller at the same exposure frequency as the near-infrared camera (multiple cameras on the VR device use the same frequency, such as 25Hz, meaning a command is sent every 40ms). This command carries information such as the current exposure time and the LED ring's illumination. Upon receiving the exposure command, the VR controller illuminates the LED ring when the timestamp of the current exposure time arrives, allowing the VR device's camera to expose and image the LED ring on the controller. Based on this image, the controller's 6DoF pose in the world coordinate system is calculated.

[0067] S403: Repeatedly send the first message responding to the exposure command to the VR device until a response to the first message is received from the VR device.

[0068] Based on the received exposure command, the VR controller can obtain the current exposure time of the camera on the VR device and send the 3Dof pose of the VR controller at the current exposure time to the VR device in the first message.

[0069] Since the timelines of the VR device and the VR controller are synchronized, when the camera on the VR device exposes the LED ring on the VR controller for the first time, the VR controller can obtain the exposure time of the next frame of the controller image. Based on the 6DoF pose of the VR controller in the world coordinate system determined from the previous frame, it can predict the 6DoF pose of the VR controller in the world coordinate system at the exposure time of the next frame. Therefore, for the first exposure of the field, when the VR controller sends its first message to the VR device, it can also carry the predicted 6DoF pose of the VR controller in the world coordinate system at the current exposure time.

[0070] Among them, the 3Dof pose and 6Dof predicted pose in the first message are used to determine the 2D-3D mapping relationship between the LED spot and the LED light. For the specific determination process, please refer to the implementation on the VR device side.

[0071] In one example, to prevent data loss, the VR controller repeatedly sends the first message to the VR device via wireless communication before the next exposure time until it receives feedback from the VR device on the first message, so as to ensure that the VR device receives the 3DoF pose and the 6DoF predicted pose.

[0072] Compared to VR controllers in camera technology that transmit motion data collected by their built-in IMU to VR devices at high frequencies, in this embodiment, the VR controller only needs to transmit the pose calculation results (3Dof pose) and pose prediction results (6Dof pose) at the exposure time to the VR device at low frequencies. This reduces the pressure on the transmission bandwidth and is less affected by electromagnetic interference, channel occupancy, and frequency band occupancy. Furthermore, since the transmission frequency of the pose results is low, the data loss problem can be solved by repeated transmission, which helps to improve positioning accuracy and robustness.

[0073] S404: Receive data packets repeatedly sent by the VR device before calculating the next 6DoF true pose of the VR controller in the world coordinate system.

[0074] In one example, the calculation of the VR controller's 6DoF pose in the world coordinate system is deployed on the VR device. The VR device calculates the VR controller's initial 6DoF pose in the world coordinate system based on the 3DoF pose and 6DoF predicted pose in the first message sent by the VR controller in response to the exposure command, combined with the controller image captured at the current exposure moment when the LED ring is lit, and the VR device's own 6DoF pose in the camera coordinate system output by the Simultaneous Localization and Mapping (SLAM) system.

[0075] After calculating the VR controller's initial 6DOf pose in the world coordinate system at the current exposure time, the VR device encapsulates the 2D-3D mapping relationship between the LED spot and the LED light obtained during the calculation process, the 2D coordinates of each LED spot in the controller image acquired at the current exposure time, and the VR controller's initial 6DOf pose in the world coordinate system at the current exposure time into a data packet, and sends it to the VR controller through wireless communication technology. This allows the VR controller to combine the motion data acquired by the built-in IMU before receiving the data packet to perform nonlinear optimization of the VR controller's initial 6DOf pose.

[0076] In one example, to ensure the effective transmission of the initial 6DoF pose corresponding to the current frame controller image and to prevent data loss caused by electromagnetic interference, channel occupancy, frequency band occupancy, etc., the VR device will repeatedly send the data packet corresponding to the current exposure time to the VR controller before the VR device calculates the next 6DoF true pose of the VR controller in the world coordinate system.

[0077] S405: Based on the received data packet and the motion data collected by the built-in IMU before the data packet was received, perform nonlinear optimization of the VR controller's initial 6DoF pose in the world coordinate system.

[0078] In one example, the VR controller receives a data packet from the VR device and combines it with motion data acquired by the built-in IMU before receiving the data packet (i.e., between the current exposure time and the data packet reception time). Using the open-source Ceres math library, the reprojection error of each LED spot on the controller image is calculated, and the IMU integral error is calculated for nonlinear optimization. The intrinsic parameters of the VR controller's built-in IMU (including the offset of the accelerometer and angular velocity in three axes, scale factor, and misalignment matrix (i.e., the rotation matrix for the transformation between the accelerometer coordinate system and the angular velocity coordinate system) and the optimized 6DoF target pose of the VR controller in the world coordinate system are obtained.

[0079] S406: Based on the optimized 6DoF target pose, determine the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering moment.

[0080] In one example, after the VR controller obtains the optimized 6DoF target pose, a deep learning algorithm is used to determine the VR controller's predicted 6DoF pose in the world coordinate system at the current rendering moment based on the optimized 6DoF target pose from the most recent exposure (usually the current exposure time).

[0081] S407: Receives rendering instructions sent by the VR device.

[0082] To provide users with a more intuitive experience of interacting with a 3D virtual scene, a 3D model can be rendered and displayed in the virtual environment based on the VR controller's 6DoF (6-DoF) actual pose. Therefore, in one example, the VR device can send rendering commands to the VR controller based on the rendering frequency of its built-in display (e.g., 72Hz, i.e., once every 13.9ms) to request the VR controller's 6DoF predicted pose at the current rendering moment. These rendering commands contain information about the current rendering time.

[0083] S408: Send a second message to the VR device in response to the rendering command. The second message carries the 6DoF predicted pose of the VR controller at the current rendering time, so that the VR device can render and display the VR controller in the current screen according to the 6DoF predicted pose of the VR controller at the current rendering time.

[0084] After receiving the rendering command, the VR controller sends its 6DoF predicted pose in the world coordinate system at the current rendering moment to the VR device via a second message. The VR device can then use filtering or interpolation algorithms to smooth the 6DoF predicted pose of the VR controller at the current rendering moment and the 6DoF predicted pose of the VR controller at the previous rendering moment, obtaining the final 6DoF predicted pose of the VR controller at the current rendering moment. Based on the final 6DoF predicted pose, the VR controller is rendered and displayed in the current screen, allowing the user to intuitively experience the three-dimensional interaction process between their hand and the virtual scene.

[0085] In one example, the VR controller can also predict the 6DoF pose at the current exposure moment and send the prediction result to the VR device to improve the computational efficiency of the 6DoF pose at the next exposure moment. Specifically, the VR controller obtains its target 6DoF pose in the world coordinate system by performing nonlinear optimization on its initial 6DoF pose in the world coordinate system at the current exposure moment. Using a deep learning algorithm, it determines the predicted 6DoF pose of the VR controller in the world coordinate system at the next exposure moment based on the target 6DoF pose of the VR controller in the world coordinate system at the current exposure moment. To prevent data loss, before the next exposure moment arrives, the VR controller repeatedly sends messages to the VR device to transmit the predicted 6DoF pose for the next exposure moment. In this way, when the camera captures the controller image at the next exposure moment, the VR device can estimate the area of ​​the VR controller in the controller image based on the predicted 6DoF pose, achieving rapid detection of the VR controller and thus improving the computational efficiency of the VR controller's true 6DoF pose in the world coordinate system at the next exposure moment.

[0086] In one example, after the VR controller and VR device are synchronized on the timeline, the VR controller can obtain not only the next exposure time of the VR device but also the next rendering time. This allows it to predict the 6DoF pose of the VR controller in the world coordinate system at the next rendering time and send the prediction result to the VR device, thus smoothing the VR controller's 6DoF pose and reducing jitter in the virtual image. Specifically, after determining the VR controller's predicted 6DoF pose in the world coordinate system at the current rendering time, a deep learning algorithm is used to determine the predicted 6DoF pose of the VR controller in the world coordinate system at the next rendering time. To prevent data loss, before the next rendering time arrives, the VR controller repeatedly sends messages to the VR device to transmit the predicted 6DoF pose for the next rendering time, thereby achieving smoothing of the VR controller's 6DoF pose.

[0087] In the embodiments of this application, the format of each single message sent by the VR controller to the VR device is as follows: Figure 5As shown, in addition to the message header and message trailer, it also includes Part1-Part9 fields. Among them:

[0088] Part 1 indicates the length of this message;

[0089] Part 2 contains the timestamp of the most recent LED ring data received. The LED ring data includes the 2D coordinates of each LED spot in the most recently exposed handle image, the 2D-3D mapping relationship between the LED spot and the LED light, etc.

[0090] Part 3 contains the 3Dof pose timestamps of the VR controllers;

[0091] Part 4 is the 3Dof pose;

[0092] Part 5 is the timestamp of the current exposure moment;

[0093] Part 6 shows the 6DoF predicted pose of the VR controller in the world coordinate system at the current exposure moment.

[0094] Part 7 is the timestamp of the current rendering moment;

[0095] Part 8 shows the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering moment.

[0096] Part 9 is the verification code for this message.

[0097] like Figure 6 The diagram shown is a flowchart of the VR device implementation process in the VR controller positioning algorithm. The process mainly includes the following steps:

[0098] S601: Sends exposure commands to the VR controller according to the camera's exposure frequency, obtains the controller image captured by the camera when the LED ring is lit at the current exposure moment, and performs LED spot detection on the controller image to obtain the 2D coordinates of each LED spot.

[0099] The LED ring on the VR controller is lit in a flashing manner. In order to ensure that the camera of the VR device can capture the LED ring, the VR device can send an exposure command to the VR controller according to the camera's exposure frequency, so as to notify the VR controller to light up the LED ring at the current exposure time indicated by the exposure command, and perform infrared imaging when the LED ring is lit to obtain the controller image.

[0100] Furthermore, the VR device uses deep learning algorithms or traditional clustering algorithms to identify the controller image at the current exposure moment, segment the controller region, and perform LED spot detection within the controller region to obtain the 2D coordinates of each LED spot in the controller image.

[0101] S602: Receives the first message sent by the VR controller in response to the exposure command.

[0102] The first message is sent repeatedly at low frequency by the VR controller, containing the 3Dof pose of the VR controller at the current exposure time and the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time.

[0103] S603: Determine if the handle image is the first frame exposure image. If yes, execute S604; otherwise, execute S605.

[0104] The 3D structure of the LED light ring on the VR controller is known, but which LED spot in the controller image corresponds to which LED in the LED light ring on the VR controller is unknown. Depending on whether the controller image is the first frame of exposure, different methods can be used to establish the 2D-3D mapping relationship between the two-dimensional LED spot and the three-dimensional LED.

[0105] In one example, the 2D-3D mapping relationship between the LED spot and the LED light can be represented by the 2D coordinates of the LED spot and the identifier of the LED light, such as {ID: (x, y)}, where ID represents the identifier of the LED light in the LED light ring, and (x, y) represents the 2D coordinates of the LED spot formed by the LED light in the handle image.

[0106] S604: Determine the 2D-3D mapping relationship between the LED spot and the LED light based on the 3Dof pose of the VR controller at the current exposure time.

[0107] When the handle image is the first frame of exposure, the process for determining the 2D-3D mapping relationship between the LED spot and the LED light is described in [reference needed]. Figure 7 It mainly includes the following steps:

[0108] S6041: Based on the 2D coordinates of each LED spot and the 3D structure of the LED ring, determine the 2D-3D mapping relationship between multiple sets of LED spots and LED lights.

[0109] In one example, the VR device analyzes the 3D structure of the LED ring based on the 2D coordinates of each LED spot in the controller image, and determines the possible LED light corresponding to each LED spot through an enumeration algorithm, thereby obtaining multiple sets of 2D-3D mapping relationships between LED spots and LED lights.

[0110] For each group of LED light spots and the 2D-3D mapping relationship between LED lights, execute steps S6042 to S6044.

[0111] S6042: Obtain the 3D coordinates of each LED spot corresponding to its respective LED light, and combine them with the 2D coordinates of each LED spot and the pre-calibrated camera parameters to determine the 6Dof pose of the VR controller in the camera coordinate system.

[0112] In practice, based on the 2D-3D mapping relationship between LED spots and LED lights, the identifier of each LED spot corresponding to the LED light is determined. Based on each identifier, the 3D coordinates of the corresponding LED spot and the corresponding LED light are obtained. Then, using the PnP or P3P algorithm, based on the 2D coordinates of each LED spot and the 3D coordinates of the corresponding LED light, the 6DoF pose of the VR controller in the camera coordinate system at the current exposure time is calculated under this set of 2D-3D mapping relationships. This includes a 3D coordinate representing the position of the VR controller in the camera coordinate system and a 3D rotational attitude represented by a quaternion.

[0113] S6043: Based on its own 6DOf pose in the camera coordinate system and the pre-calibrated extrinsic parameters between the camera and the built-in IMU, the VR device obtains the 6DOf pose in the world coordinate system at the current exposure moment.

[0114] In SLAM systems, the world coordinate system is typically the coordinate system of the first frame of IMU data, and the extrinsic parameters between the camera and the built-in IMU on the VR device can be obtained through pre-calibration. Therefore, by using the head motion data acquired through its own SLAM system and the built-in IMU, and leveraging the extrinsic parameters between the camera and the built-in IMU, the VR device can determine its 6DoF pose in the world coordinate system at the current exposure moment. Specifically, the VR device multiplies its own 6DoF pose in the camera coordinate system by the extrinsic parameters of the camera and the built-in IMU on the VR device to obtain its 6DoF pose in the world coordinate system at the current exposure moment.

[0115] S6044: Determine the initial 6DoF pose of the VR controller in the world coordinate system at the current exposure time, based on the 6DoF pose of the VR device in the world coordinate system and the 6DOf pose of the VR controller in the camera coordinate system.

[0116] In one example, the VR device multiplies its 6DoF pose in the world coordinate system at the current exposure time by the 6DoF pose of the VR controller in the camera coordinate system at the current exposure time to obtain the true 6DoF pose of the VR controller in the world coordinate system at the current exposure time under this 2D-3D mapping relationship.

[0117] S6045: Use the 3DoF pose of the VR controller to verify the 6DoF initial poses corresponding to multiple sets of 2D-3D mapping relationships, and select a set of 2D-3D mapping relationships.

[0118] Since the VR controller's initial 6DoF pose in the world coordinate system at the current exposure moment includes a 3D coordinate representing the VR controller's position in the world coordinate system and a 3D rotational pose represented by a quaternion, and the first message sent by the VR controller to the VR device contains a 3D rotational pose represented by a quaternion, the 6DoF initial pose corresponding to each 2D-3D mapping relationship can be verified using the 3DoF pose at the current exposure moment contained in the first message. The 2D-3D mapping relationship corresponding to the 6DoF initial pose with the smallest 3D rotational pose error at the current exposure moment is selected as the final 2D-3D mapping relationship. Specifically, using the transformation relationship between quaternions and rotation matrices, the 3D rotational pose of the VR controller in the world coordinate system at the current exposure moment is converted into a 3*3 first rotation matrix. At the same time, the 3D rotational pose of the VR controller in the 3Dof pose at the current exposure moment in the first message is also converted into a 3*3 second rotation matrix. The third row of the first rotation matrix is ​​extracted to obtain a 3*1 first vector, and the third row of the second rotation matrix is ​​extracted to obtain a 3*1 second vector. The first vector and the second vector are multiplied by a dot product. The closer the dot product result is to 1, the more accurate the 2D-3D mapping relationship corresponding to the initial 6DoF pose is. Therefore, the 2D-3D mapping relationship closest to 1 is selected from multiple sets of calculation results as the final 2D-3D mapping relationship.

[0119] S605: Determine the 2D-3D mapping relationship between the LED spot and the LED light based on the VR controller's 6DoF predicted pose in the world coordinate system.

[0120] When the handle image is a non-frame exposure image, the process for determining the 2D-3D mapping relationship between the LED spot and the LED light is described in [reference needed]. Figure 8 It mainly includes the following steps:

[0121] S6051: Based on the VR controller's 6DoF predicted pose in the world coordinate system at the current exposure time, and the pre-calibrated camera intra-parameters, project the LED ring onto the controller image at the current exposure time.

[0122] In one example, the 3D structure of the LED ring of the VR controller is known, and the 6DoF predicted pose of the VR controller in the world coordinate system at the current exposure time is also known. Using the camera's intrinsic parameters, the three-dimensional LED ring can be projected onto the two-dimensional controller image.

[0123] S6052: Based on the projection coordinates of each LED spot in the handle image and the 2D coordinates of each LED spot in the handle image, directly determine a set of 2D-3D mapping relationships between LED spots and LED lights.

[0124] In one example, for each LED spot, the LED corresponding to the projection coordinates that are closest to its 2D coordinates in the handle image is taken as the LED corresponding to that LED spot, thus obtaining the 2D-3D mapping relationship between the LED spot and the LED.

[0125] Compared to Figure 7 The process of determining the 2D-3D mapping relationship shown reduces the verification and screening process of the 2D-3D mapping relationship and improves the efficiency of pose calculation.

[0126] In one example, when the error between the projected coordinates of the LED light and the actual 2D coordinates of the LED spot is large, the determined mapping relationship may be inaccurate. In this case, a more precise approach can be adopted. Figure 7 The method shown determines the 2D-3D mapping relationship between the LED spot and the LED lamp.

[0127] S606: Based on the 2D-3D mapping relationship between the LED spot and the LED light and the 2D coordinates of each LED spot, determine the initial 6Dof pose of the VR controller in the world coordinate system at the current exposure moment.

[0128] In practice, the VR device obtains the 3D coordinates of the LED light corresponding to each LED light spot based on the 2D-3D mapping relationship between the LED light spot and the LED light. Combined with the 2D coordinates of each LED light spot, the PnP algorithm is used to calculate the initial 6DoF pose of the VR controller in the world coordinate system at the current exposure time.

[0129] S607: Encapsulate the 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot at the current exposure time, and the initial 6DoF pose of the VR controller in the world coordinate system into a data packet, and repeatedly send it to the VR controller before calculating the next initial 6DoF pose of the VR controller in the world coordinate system.

[0130] After calculating the VR controller's initial 6DOf pose in the world coordinate system at the current exposure moment, the VR device encapsulates the 2D-3D mapping relationship between the LED spot and the LED light obtained during the calculation process, the 2D coordinates of each LED spot in the controller image acquired at the current exposure moment, and the VR controller's initial 6DOf pose in the world coordinate system at the current exposure moment into a data packet. To prevent data loss, before calculating the next initial 6DOf pose of the VR controller in the world coordinate system, the VR device repeatedly sends this data packet to the VR controller via wireless communication technology, so that the VR controller can combine the motion data acquired by the built-in IMU to perform nonlinear optimization of the VR controller's initial 6DOf pose.

[0131] S608: Send rendering commands to the VR controller according to the rendering frequency to obtain the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering moment indicated by the rendering commands.

[0132] To allow users to more intuitively experience the interaction between their hands and the 3D virtual scene, a 3D model can be rendered and displayed in the virtual screen based on the 6DoF pose of the VR controller at the current rendering moment. Therefore, in one example, the VR device can send rendering commands to the VR controller to request the 6DoF pose of the VR controller at the current rendering moment, based on the rendering frequency of the built-in display (e.g., 72Hz, i.e., once every 13.9ms).

[0133] S609: Receives the second message sent by the VR controller in response to the rendering command, and renders and displays the VR controller in the current screen according to the 6DoF predicted pose of the VR controller at the current rendering time carried in the second message.

[0134] Upon receiving the rendering command, the VR controller is optimized based on its 6DoF target pose in the world coordinate system at the current exposure moment. The system then estimates the predicted 6DoF pose of the VR controller in the world coordinate system at the current rendering moment and transmits this predicted 6DoF pose to the VR controller via a second message. The VR device, based on the predicted 6DoF pose of the VR controller in the world coordinate system at the current rendering moment, renders and displays the VR controller in the current screen using its graphics rendering engine, allowing the user to intuitively experience the 3D interaction between their hands and the virtual scene.

[0135] In the embodiments of this application, the format of a single instruction sent by the VR device to the VR controller each time is as follows: Figure 9 As shown, in addition to the message header and message trailer, it also includes Part1-Part13 fields. Among them:

[0136] Part 1 indicates the length of this message;

[0137] Part 2 indicates whether the LED ring is lit. When the LED is lit (i.e., Part 3 is "Yes"), Part 3 is reset to "No".

[0138] Part 3 indicates whether the 3Dof pose of the VR controller has been received during the LED ring illumination cycle;

[0139] Part 4 is the timestamp of the current exposure moment;

[0140] Part 5 is the timestamp of the current rendering moment;

[0141] Part 6 is the timestamp for calculating the 6DoF true pose of the VR controller at the moment of the most recent exposure;

[0142] Part 7 is the 6DoF true pose of the VR controller at the previous exposure time. If no 6DoF true pose of the VR controller has been calculated, the value of this field is marked as 0.

[0143] Part8 is the first sequence number of the camera that observed the most LED spots. If the VR controller is not observed by any camera, the value of this field is set to 0.

[0144] Part 9 shows the number of LED spots observed by the camera with the first number;

[0145] Part10 is the second sequence number of the camera that observed the second most LED spot. If the VR controller is not observed by any camera or is only observed by one camera, the value of this field is marked as 0.

[0146] Part 11 represents the number of LED spots observed by the second camera;

[0147] Part 12 is a variable-length field that sequentially describes the 2D coordinates of each LED spot observed by the first and second numbered cameras. If no LED spot is observed by any camera, this field is empty.

[0148] Part 13 is the verification code for this message.

[0149] In the absence of electromagnetic interference, signal occupancy, frequency band occupancy, etc., the uplink and downlink communication mechanism between VR controllers and VR devices is as follows: Figure 10 As shown, for each command sent by the VR device, the VR controller returns a message. Under conditions affected by electromagnetic interference, signal occupancy, frequency band occupancy, etc., the uplink and downlink communication mechanisms between the VR device and the VR controller are as follows: Figure 11 As shown. Wherein:

[0150] The dashed line represents the exposure command sent to the VR controller when the VR device exposes and images the LED ring of the VR controller according to a fixed camera exposure frequency. The exposure command means to indicate the timestamp of the camera exposure and imaging of the VR device to the VR controller and instruct the VR controller to light up the LED ring. The contents of each field in the exposure command are shown in Table 1.

[0151] Table 1. Contents of Fields Part 1-Part 13 in Exposure Instructions Issued by VR Devices

[0152] Part 1 L1 Part 8 0 Part 2 yes Part 9 0 Part 3 no Part 10 0 Part 4 The timestamp T0 of the current exposure moment Part 11 0 Part 5 0 Part 12 null Part 6 0 Part 13 Verification code 1 Part 7 0

[0153] After receiving the exposure command, the VR controller immediately replies with a message (represented by a solid line). This message contains the current exposure time when the LED ring is lit as indicated by the exposure command, the 3Dof pose of the VR controller calculated based on the motion data collected by the built-in IMU, and the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time (if a prediction result exists). The contents of fields Part1-Part9 in this message are shown in Table 2.

[0154] Table 2. Contents of Part 1-Part 9 in the response message corresponding to the exposure command.

[0155]

[0156] After receiving the response message from the VR controller in response to the exposure command, the VR device compares the timestamp of the camera exposure image in the sent exposure command. If the timestamp of the 3Dof pose in the response message matches the timestamp in the exposure command, the VR device replies with an acknowledgment message (represented by a dotted line) to notify the VR controller that it has received the data returned by the VR controller. The contents of fields Part1-Part13 in the acknowledgment message sent by the VR device are shown in Table 3.

[0157] Table 3. Contents of Part 1-Part 13 fields in the confirmation message sent by the VR device

[0158] Part 1 L3 Part 8 0 Part 2 no Part 9 0 Part 3 yes Part 10 0 Part 4 The timestamp T0 of the current exposure moment Part 11 0 Part 5 0 Part 12 null Part 6 0 Part 13 Check code 3 Part 7 0

[0159] The double-dotted line represents the rendering command issued by the VR device to the VR controller according to the rendering frequency of the display. The rendering command includes the timestamp of the current rendering moment to request the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering moment. The contents of the Part1-Part13 fields in the rendering command are shown in Table 4.

[0160] Table 4. Contents of Part 1-Part 13 fields in the rendering instructions issued by the VR device.

[0161] Part 1 L4 Part 8 0 Part 2 no Part 9 0 Part 3 yes Part 10 0 Part 4 0 Part 11 0 Part 5 The timestamp T1 of the current rendering moment Part 12 null Part 6 0 Part 13 Check code 4 Part 7 0

[0162] After receiving the rendering command from the VR device, the VR controller immediately replies with a message (indicated by a solid line) stating that it has sent the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering moment. The contents of the Part1-Part9 fields in the reply message to the rendering command are shown in Table 5.

[0163] Table 5. Contents of Part 1-Part 9 in the response message corresponding to the rendering command.

[0164]

[0165] The long and short dashed lines represent the data packets sent by the VR device to the VR controller. These packets include the VR controller's initial 6DoF pose in the world coordinate system calculated based on the controller image captured at the current exposure time, the 2D coordinates of each LED spot in the controller image, and the 2D-3D mapping relationship between the LED spots and the LED lights. This allows the VR controller to perform nonlinear optimization of its initial 6DoF pose in the world coordinate system at the current exposure time by combining motion data collected by its built-in IMU, thereby obtaining the target 6DoF pose.

[0166] It should be noted that the data packet delivery frequency is not strictly fixed; that is, the time interval between data packets corresponding to two frames of controller images is not fixed. Based on the computing power of the VR device, the data packet delivery frequency is close to the exposure frequency. The contents of Part1-Part13 fields in the message that delivers this data packet are shown in Table 6.

[0167] Table 6. Contents of Part 1-Part 13 fields in the data packet message sent by the VR device

[0168]

[0169] After receiving this message, the VR controller immediately replies with a message (indicated by a solid line) to notify the VR device that it has received the data packet. The contents of the Part1-Part9 fields in the VR controller's reply message to the VR device for the data packet are shown in Table 7.

[0170] Table 7. Contents of Part 1-Part 9 in the response message corresponding to the rendering command.

[0171]

[0172]

[0173] In VR controller positioning methods that coordinate with VR devices, to avoid electromagnetic interference, signal occupancy, and frequency band occupancy, and to ensure the accuracy and robustness of the positioning results, it is necessary to ensure the effective transmission of the VR controller's response messages to exposure commands issued by the VR device and the messages from the VR device to the VR controller during communication between the VR device and the VR controller. In the embodiments of this application, the VR controller and VR headset only need to transmit exposure and rendering time data at low frequency. When the usage environment of the VR all-in-one device affects the transmission of these two messages, data retransmission can be guaranteed within a limited time, thereby ensuring the positioning accuracy and robustness of the VR controller. Specifically, as shown... Figure 11As shown, in response to exposure commands, when the VR controller does not receive a confirmation message represented by a single dotted line, it will repeatedly send a certain number of response messages; in response to data packets sent by the VR device, before the next 6DoF initial pose is calculated, it will repeatedly send a certain number of messages represented by long and short dashed lines.

[0174] It should be noted that when there is no conflict in the contents of the Part1-Part12 fields between the instructions and messages sent by the VR device to the VR controller, multiple types of instructions and messages can be merged into one message and sent. When there is a conflict in the contents of the Part1-Part12 fields between the instructions and messages sent by the VR device to the VR controller, they can be sent in sequence according to the exposure rendering order.

[0175] like Figure 12 As shown, in the VR controller positioning algorithm provided in this application embodiment, the specific interaction process between the VR device and the VR controller mainly includes the following steps:

[0176] S1201: The VR controller calculates its 3Dof pose in real time based on motion data continuously collected by the built-in IMU.

[0177] S1202: The VR device sends an exposure command to the VR controller based on the camera's exposure frequency. The exposure command includes the camera's current exposure time and is used to instruct the VR controller to light up the LED ring at the current exposure time.

[0178] S1203~S1204: The VR controller lights up the LED ring according to the exposure command, and in response to the exposure command, repeatedly sends a first message to the VR device containing the 3Dof pose and 6Dof predicted pose of the VR controller at the current exposure time.

[0179] S1205: VR device captures the controller image when the LED ring is lit.

[0180] S1206: After receiving the first message, the VR device returns an acknowledgment message to the VR controller to notify the VR controller that it has received the pose data of the current exposure moment sent by the VR controller.

[0181] S1207: The VR device performs spot detection on the controller image captured at the current exposure time and obtains the 2D coordinates of each spot.

[0182] S1208: The VR device determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 3Dof pose or 6Dof predicted pose of the VR controller at the current exposure time.

[0183] S1209: The VR device determines the initial 6DoF pose of the VR controller in the world coordinate system based on the 2D-3D mapping relationship between the LED spot and the LED light, as well as the 2D coordinates of each spot.

[0184] S1210: The VR device packages the 2D-3D mapping relationship between LED light spots and LED lights, the 2D coordinates of each light spot, and the initial 6DoF pose of the VR controller into a data packet, and sends it to the VR controller via a message.

[0185] S1211: The VR controller performs nonlinear optimization on the received 6DoF initial pose based on the motion data collected by the built-in IMU before receiving the data packet, and obtains the 6DoF target pose.

[0186] S1212: The VR device sends rendering instructions to the VR device according to the display's rendering frequency. The rendering instructions carry the current rendering time.

[0187] S1213: The VR controller responds to the rendering command and sends the 6DoF predicted pose in the world coordinate system at the current rendering time indicated by the rendering command to the VR device through a second message. The 6DoF predicted pose is predicted by the VR controller based on the 6DoF target pose optimized by the most recent exposure time.

[0188] S1214: The VR device predicts the pose of the VR controllers in the world coordinate system based on the current rendering time's VR controller pose and renders and displays the VR controllers in the current virtual screen.

[0189] S1215: The VR controller predicts the 6DoF target pose based on the optimized 6DoF target pose, and obtains the 6DoF predicted pose of the controller in the world coordinate system when the next frame of the controller image is exposed.

[0190] S1216: Before calculating the next 6DoF initial pose, the VR controller repeatedly sends the 6DoF predicted pose of the next frame of the controller image to the VR device so that the VR device can perform LED spot detection based on the 6DoF predicted pose.

[0191] S1217: The VR device sends rendering instructions to the VR device according to the display's rendering frequency. The rendering instructions carry the next rendering time.

[0192] S1218: The VR controller responds to the rendering command and sends the 6DoF predicted pose in the world coordinate system at the next rendering moment indicated by the rendering command to the VR device through the second message. The 6DoF predicted pose is predicted by the VR controller based on the 6DoF target pose optimized by the most recent exposure moment.

[0193] S1219: The VR device renders and displays the VR controllers in the next virtual scene based on the VR controllers' predicted pose in the world coordinate system at the next rendering time (6DoF).

[0194] This application embodiment, based on near-infrared spot projection positioning technology, nonlinear optimization technology, and wireless device time synchronization technology, redesigns the system architecture for calculating the 6DOf pose of VR controllers, specifically for the use scenario of VR all-in-one devices. The 6DOf positioning of the VR controllers is jointly achieved by the VR controllers and the VR device. Specifically, the 3Dof pose calculation, nonlinear 6Dof pose optimization calculation, and 6Dof pose prediction calculation processes are deployed on the VR controller side, while the processes of exposure imaging, LED spot detection, establishing the 2D-3D mapping relationship between the spot and the LED light, and the initial 6Dof pose calculation of the VR controllers are deployed on the VR device side. Thus, the VR... The VR controller does not need to transmit the motion data collected in real time by its built-in IMU to the VR device at high frequency. Instead, it only needs to transmit the pose calculation results (3DoF pose and 6DOF predicted pose) at the required time (including exposure time and rendering time) to the VR device at low frequency. When there are problems such as electromagnetic interference, channel occupation, and frequency band occupation in the VR all-in-one device's usage environment, the VR controller can ensure that the data is not lost through data retransmission, thereby preventing the impact of data loss on the VR controller's 6DoF pose calculation, improving positioning accuracy and robustness, reducing the pressure on transmission bandwidth, improving positioning efficiency, and effectively expanding the breadth of VR all-in-one application scenarios.

[0195] Based on the same technical concept, this application provides a VR controller that can work with a VR device that is synchronized with the timeline to implement the above-mentioned VR controller positioning method and achieve the same technical effect.

[0196] See Figure 13 The VR controller includes a near-infrared LED light ring 1301, an IMU 1302, a memory 1303, a processor 1304, and a communication interface 1305. The near-infrared LED light ring 1301, the IMU 1302, the memory 1303, the processor 1304, and the communication interface 1305 are connected via a bus 1306.

[0197] The memory 1303 stores a computer program, and the processor 1304 performs the following operations according to the computer program:

[0198] Based on the motion data continuously acquired by the IMU1302, the 3Dof pose of the device is calculated in real time.

[0199] The VR device receives the exposure command sent by the VR device according to the camera exposure frequency through the communication interface 1305, and lights up the near-infrared LED ring 1301 according to the current exposure time indicated by the exposure command.

[0200] Through the communication interface 1305, the first message responding to the exposure command is repeatedly sent to the VR device until the feedback of the VR device to the first message is received; wherein, the first message includes the 3Dof pose of the VR controller at the current exposure time and the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time, and the 3Dof pose and the 6Dof predicted pose are used to determine the 2D-3D mapping relationship between the LED spot and the LED light.

[0201] Through the communication interface 1305, the VR device receives data packets repeatedly sent before calculating the next 6DOf real pose. The data packets include the 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot in the controller image acquired at the current exposure time, and the initial 6DOf pose of the VR controller in the world coordinate system at the current exposure time.

[0202] Through the communication interface 1305, based on the received data packet and the motion data collected by the IMU before the data packet was received, the initial 6DoF pose of the VR controller in the world coordinate system is nonlinearly optimized to obtain the target 6DoF pose.

[0203] Based on the target 6DoF pose, determine the predicted 6DoF pose of the VR controller in the world coordinate system at the current rendering moment;

[0204] The VR device receives rendering instructions from the VR device via the communication interface 1305 and sends a second message in response to the rendering instructions to the VR device. The second message carries the 6DoF predicted pose so that the VR device renders and displays the VR controller in the current screen according to the 6DoF predicted pose.

[0205] Optionally, after obtaining the 6DOF target pose, the processor 1304 also executes:

[0206] Based on the 6DoF target pose of the VR controller in the world coordinate system at the current exposure time, determine the 6DoF predicted pose of the VR controller in the world coordinate system at the next exposure time;

[0207] Before the next exposure time arrives, the 6DoF predicted pose of the VR controller at the next exposure time is repeatedly sent to the VR device so that the VR device can determine the area of ​​the VR controller in the controller image at the next exposure time.

[0208] After the optional processor 1304 determines the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering moment, it also performs:

[0209] Based on the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering time, determine the 6DoF predicted pose of the VR controller in the world coordinate system at the next rendering time.

[0210] Before the next rendering time arrives, the 6DoF predicted pose of the VR controller for the next rendering time is repeatedly sent to the VR device to achieve pose smoothing.

[0211] Optionally, the message sent by the VR controller to the VR device includes at least the message length, the timestamp of the most recent receipt of LED ring data, the timestamp of the 3Dof pose, the 3Dof pose, the timestamp of the current exposure moment, the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure moment, the timestamp of the current rendering moment, the 6Dof pose of the VR controller in the world coordinate system at the current rendering moment, and a checksum field.

[0212] It should be noted that, Figure 13 This is merely an example illustrating the hardware necessary for a VR controller to perform the VR controller positioning method steps provided in this application embodiment. Not shown, the VR controller may also include conventional VR controller hardware such as trigger buttons, joysticks, and buttons.

[0213] Based on the same technical concept, this application provides a VR device that can work with a VR controller synchronized with the timeline to implement the above-mentioned VR controller positioning method and achieve the same technical effect.

[0214] See Figure 14 The VR device includes multiple near-infrared cameras 1401, a display 1402, a memory 1403, a processor 1404, and a communication interface 1405. The multiple near-infrared cameras 1401, the display 1402, the memory 1403, the processor 1404, and the communication interface 1405 are connected via a bus 1406.

[0215] The memory 1403 stores a computer program, and the processor 1404 performs the following operations according to the computer program:

[0216] Through the communication interface 1405, an exposure command is sent to the VR controller according to the exposure frequency of the near-infrared camera 1401, the controller image when the LED ring is lit at the current exposure moment is obtained, and the LED spot detection is performed on the controller image to obtain the 2D coordinates of each LED spot;

[0217] The communication interface 1405 receives a first message sent by the VR controller in response to the exposure command. If the controller image is the first frame exposure image, the 2D-3D mapping relationship between the LED spot and the LED light is determined according to the 3Dof pose of the VR controller at the current exposure time contained in the first message. Otherwise, the 2D-3D mapping relationship between the LED spot and the LED light is determined according to the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time contained in the first message.

[0218] Based on the 2D-3D mapping relationship between the LED spot and the LED light and the 2D coordinates of each LED spot, the initial 6Dof pose of the VR controller in the world coordinate system at the current exposure time is determined;

[0219] Before the next 6DoF initial pose calculation, the 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot, and the 6DoF initial pose of the VR controller are encapsulated into a data packet and repeatedly sent to the VR controller through the communication interface 1405.

[0220] Through the communication interface 1405, rendering commands are sent to the VR controller according to the rendering frequency of the display 1402;

[0221] The communication interface 1405 receives a second message from the VR controller in response to the exposure command. Based on the 6DoF predicted pose of the VR controller at the current rendering time carried in the second message, the VR controller is rendered in the current screen and displayed on the monitor.

[0222] Optionally, the processor 1404 determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 3Dof pose of the VR controller at the current exposure time contained in the first message. Specifically, the operation is as follows:

[0223] Based on the 2D coordinates of each LED spot and the 3D structure of the LED ring, determine the 2D-3D mapping relationship between multiple sets of LED spots and LED lights;

[0224] For each 2D-3D mapping relationship, perform the following operations:

[0225] Obtain the 3D coordinates of the LED light corresponding to each LED spot, and combine the 2D coordinates of each LED spot with the pre-calibrated camera parameters to determine the 6DoF pose of the VR controller in the camera coordinate system.

[0226] Based on its own 6DOf pose in the camera coordinate system and the pre-calibrated extrinsic parameters between the camera and the built-in IMU, the VR device's 6DOf pose in the world coordinate system at the current exposure time is obtained.

[0227] Based on the 6Dof pose of the VR device in the world coordinate system and the 6DOf pose of the VR controller in the camera coordinate system at the current exposure time, determine the initial 6Dof pose of the VR controller in the world coordinate system at the current exposure time.

[0228] The 3DoF pose of the VR controller is used to verify the 6DoF initial poses corresponding to multiple sets of 2D-3D mapping relationships, and a set of 2D-3D mapping relationships is selected.

[0229] Optionally, the processor 1404 determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 6DoF predicted pose of the VR controller in the world coordinate system at the current exposure time contained in the first message. Specifically, the operation is as follows:

[0230] Based on the 6DoF predicted pose of the VR controller in the world coordinate system at the current exposure time, and the pre-calibrated camera intra-parameters, the LED light ring is projected onto the controller image at the current exposure time.

[0231] Based on the projection coordinates of each LED spot in the handle image and the 2D coordinates of each LED spot in the handle image, a set of 2D-3D mapping relationships between LED spots and LED lights are directly determined.

[0232] Optionally, the instruction sent by the VR device to the VR controller includes at least the message length, a flag indicating whether the LED ring is lit, a flag indicating whether the 3Dof pose of the VR controller was received within the LED ring lighting cycle, a timestamp of the current exposure time, a timestamp of the current rendering time, a timestamp of the 6Dof initial pose of the VR controller calculated at the most recent exposure time, the 6Dof target pose of the VR controller at the previous exposure time, the first sequence number of the camera that observed the most LED spots, the number of LED spots observed by the first sequence number camera, the second sequence number of the camera that observed the second most LED spots, the number of LED spots observed by the second sequence number camera, and the 2D coordinates and checksum fields of each LED spot observed by the first sequence number camera and the second sequence number camera described in sequence.

[0233] It should be noted that, Figure 14 This is merely an example illustrating the hardware necessary for a VR device to perform the steps of a VR controller positioning method provided in this application embodiment. Not shown, the VR device may also include conventional hardware such as an IMU, speaker, microphone, operation keys, and power supply.

[0234] It should be noted that, Figure 13 and Figure 14 The memory in the memory can be volatile memory, such as random access memory; it can also be non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; or it can be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory can be a combination of the above-mentioned memories; the processor can include one or more central processing units or digital processing units, etc.

[0235] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform a VR controller positioning method as described in the foregoing embodiments.

[0236] This application also provides a computer program product for storing a computer program that executes a VR controller positioning method described in the foregoing embodiments.

[0237] Those skilled in the art should understand that any modifications and variations made to this application without departing from the spirit and scope of this application are within the scope of the claims of this application and their equivalents.

Claims

1. A VR controller positioning method, characterized in that, Applied to a VR controller, wherein the VR controller is synchronized with the timeline of a VR device, the method includes: The 3Dof pose of the VR controller is calculated in real time based on motion data continuously collected by the built-in IMU. Receive the exposure command sent by the VR device according to the camera exposure frequency, and light up the LED ring according to the current exposure time indicated by the exposure command; The system repeatedly sends a first message responding to the exposure command to the VR device until it receives feedback from the VR device on the first message; wherein, the first message includes the 3Dof pose of the VR controller at the current exposure time and the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time, and the 3Dof pose and the 6Dof predicted pose are used to determine the 2D-3D mapping relationship between the LED spot and the LED light. The VR device receives data packets repeatedly sent before calculating the next 6DoF true pose of the VR controller in the world coordinate system. The data packets contain the 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot in the controller image acquired at the current exposure time, and the initial 6DoF pose of the VR controller in the world coordinate system at the current exposure time. Based on the received data packet and the motion data collected by the built-in IMU before the data packet was received, the initial 6DoF pose of the VR controller in the world coordinate system is nonlinearly optimized to obtain the target 6DoF pose. Based on the target 6DoF pose, determine the predicted 6DoF pose of the VR controller in the world coordinate system at the current rendering moment; The system receives a rendering command sent by the VR device and sends a second message responding to the rendering command to the VR device. The second message carries the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering time, so that the VR device renders and displays the VR controller in the current screen according to the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering time.

2. The method as described in claim 1, characterized in that, After obtaining the 6DOF target pose, the method further includes: Based on the 6DoF target pose of the VR controller in the world coordinate system at the current exposure time, determine the 6DoF predicted pose of the VR controller in the world coordinate system at the next exposure time; Before the next exposure time arrives, the 6DoF predicted pose of the VR controller at the next exposure time is repeatedly sent to the VR device so that the VR device can determine the area of ​​the VR controller in the controller image at the next exposure time.

3. The method as described in claim 1, characterized in that, After determining the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering moment, the method further includes: Based on the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering time, determine the 6DoF predicted pose of the VR controller in the world coordinate system at the next rendering time. Before the next rendering time arrives, the 6DoF predicted pose of the VR controller for the next rendering time is repeatedly sent to the VR device to achieve pose smoothing.

4. The method according to any one of claims 1-3, characterized in that, The format of a single message sent by the VR controller to the VR device includes at least the following fields: message length, timestamp of the most recent receipt of LED ring data, timestamp of 3Dof pose, 3Dof pose, timestamp of current exposure moment, 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure moment, timestamp of current rendering moment, 6Dof predicted pose of the VR controller in the world coordinate system at the current rendering moment, and checksum.

5. A VR controller positioning method, characterized in that, Applied to VR devices, wherein the timeline of the VR device is synchronized with that of a VR controller, the method includes: According to the camera's exposure frequency, an exposure command is sent to the VR controller to obtain the controller image captured by the camera when the LED ring is lit at the current exposure moment, and LED spot detection is performed on the controller image to obtain the 2D coordinates of each LED spot; The system receives a first message sent by the VR controller in response to the exposure command. If the controller image is the first frame exposure image, the system determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 3Dof pose of the VR controller at the current exposure time contained in the first message. Otherwise, the system determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time contained in the first message. Based on the 2D-3D mapping relationship between the LED spot and the LED light and the 2D coordinates of each LED spot, the initial 6Dof pose of the VR controller in the world coordinate system at the current exposure time is determined; The 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot, and the 6DoF initial pose of the VR controller are encapsulated into a data packet, and repeatedly sent to the VR controller before calculating the next 6DoF initial pose of the VR controller in the world coordinate system. According to the rendering frequency, a rendering command is sent to the VR controller to obtain the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering time indicated by the rendering command. The system receives a second message from the VR controller in response to the exposure command, and renders and displays the VR controller in the current screen based on the 6DoF predicted pose of the VR controller at the current rendering time carried in the second message.

6. The method as described in claim 5, characterized in that, The step of determining the 2D-3D mapping relationship between the LED spot and the LED light based on the 3Dof pose of the VR controller at the current exposure time contained in the first message includes: Based on the 2D coordinates of each LED spot and the 3D structure of the LED ring, determine the 2D-3D mapping relationship between multiple sets of LED spots and LED lights; For each 2D-3D mapping relationship, perform the following operations: Obtain the 3D coordinates of the LED light corresponding to each LED spot, and combine the 2D coordinates of each LED spot with the pre-calibrated camera parameters to determine the 6DoF pose of the VR controller in the camera coordinate system. Based on its own 6DoF pose in the camera coordinate system and the pre-calibrated extrinsic parameters between the camera and the built-in IMU, the VR device's 6DoF pose in the world coordinate system at the current exposure time is obtained. Based on the 6DoF pose of the VR device in the world coordinate system and the 6DoF pose of the VR controller in the camera coordinate system at the current exposure time, determine the initial 6DoF pose of the VR controller in the world coordinate system at the current exposure time. The 3DoF pose of the VR controller is used to verify the 6DoF initial poses corresponding to multiple sets of 2D-3D mapping relationships, and a set of 2D-3D mapping relationships is selected.

7. The method as described in claim 5, characterized in that, The step of determining the 2D-3D mapping relationship between the LED spot and the LED light based on the 6DoF predicted pose of the VR controller in the world coordinate system at the current exposure time contained in the first message includes: Based on the 6DoF predicted pose of the VR controller in the world coordinate system at the current exposure time, and the pre-calibrated camera intra-parameters, the LED light ring is projected onto the controller image at the current exposure time. Based on the projection coordinates of each LED spot in the handle image and the 2D coordinates of each LED spot in the handle image, a set of 2D-3D mapping relationships between LED spots and LED lights are directly determined.

8. The method according to any one of claims 5-7, characterized in that, The format of a single instruction sent by the VR device to the VR controller includes at least the following fields: message length, a flag indicating whether the LED ring is lit, a flag indicating whether the 3Dof pose of the VR controller was received within the LED ring lighting cycle, a timestamp of the current exposure time, a timestamp of the current rendering time, a timestamp of the 6Dof initial pose of the VR controller calculated at the most recent exposure time, the 6Dof target pose of the VR controller at the previous exposure time, the first sequence number of the camera that observed the most LED spots, the number of LED spots observed by the first sequence number camera, the second sequence number of the camera that observed the second most LED spots, the number of LED spots observed by the second sequence number camera, and the 2D coordinates and checksums of each LED spot observed by the first sequence number camera and the second sequence number camera, described in sequence.

9. A VR controller, characterized in that, The VR device and the VR controller are synchronized on the timeline. The VR controller includes a near-infrared LED light ring, an IMU, a memory, a processor, and a communication interface. The near-infrared LED light ring, the IMU, the memory, the communication interface, and the processor are connected via a bus. The memory stores a computer program, and the processor performs the following operations according to the computer program: Based on the motion data continuously collected by the IMU, the 3Dof pose of itself is calculated in real time. The system receives exposure commands from the VR device according to the camera's exposure frequency via the communication interface, and illuminates the near-infrared LED ring according to the current exposure time indicated by the exposure commands. Through the communication interface, the first message responding to the exposure command is repeatedly sent to the VR device until the feedback of the VR device to the first message is received; wherein, the first message includes the 3Dof pose of the VR controller at the current exposure time and the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time, and the 3Dof pose and the 6Dof predicted pose are used to determine the 2D-3D mapping relationship between the LED spot and the LED light. Through the communication interface, the VR device receives data packets repeatedly sent before calculating the next 6DoF true pose. The data packets include the 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot in the controller image acquired at the current exposure time, and the initial 6DoF pose of the VR controller in the world coordinate system at the current exposure time. Through the communication interface, based on the received data packet and the motion data collected by the IMU before the data packet was received, the initial 6DoF pose of the VR controller in the world coordinate system is nonlinearly optimized to obtain the target 6DoF pose. Based on the target 6DoF pose, determine the predicted 6DoF pose of the VR controller in the world coordinate system at the current rendering moment; The system receives rendering instructions from the VR device through the communication interface and sends a second message in response to the rendering instructions to the VR device. The second message carries the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering time, so that the VR device renders and displays the VR controller in the current screen according to the 6DoF predicted pose of the VR controller in the world coordinate system at the current rendering time.

10. A VR device, characterized in that, The VR device and VR controller are synchronized in time. The VR device includes multiple near-infrared cameras, a display, a memory, a processor, and a communication interface. The memory stores a computer program, and the processor performs the following operations according to the computer program: Through the communication interface, an exposure command is sent to the VR controller according to the exposure frequency of the near-infrared camera, the controller image when the LED ring is lit at the current exposure moment is obtained, and the LED spot detection is performed on the controller image to obtain the 2D coordinates of each LED spot; Through the communication interface, the system receives a first message sent by the VR controller in response to the exposure command. If the controller image is the first frame exposure image, the system determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 3Dof pose of the VR controller at the current exposure time contained in the first message. Otherwise, the system determines the 2D-3D mapping relationship between the LED spot and the LED light based on the 6Dof predicted pose of the VR controller in the world coordinate system at the current exposure time contained in the first message. Based on the 2D-3D mapping relationship between the LED spot and the LED light and the 2D coordinates of each LED spot, the initial 6Dof pose of the VR controller in the world coordinate system at the current exposure time is determined; Before the next 6DoF initial pose calculation, the 2D-3D mapping relationship between the LED spot and the LED light, the 2D coordinates of each LED spot, and the 6DoF initial pose of the VR controller are encapsulated into a data packet and repeatedly sent to the VR controller through the communication interface. Through the communication interface, rendering commands are sent to the VR controller according to the rendering frequency of the display; The system receives a second message from the VR controller in response to the exposure command via the communication interface. Based on the 6DoF predicted pose of the VR controller at the current rendering time carried in the second message, the system renders the VR controller in the current screen and displays it on the monitor.

Images