Synchronized positioning method and device of head-mounted display device and handle

Abstract

The application relates to the VR / AR technical field and provides a head-mounted display device, a hand-held handle synchronization positioning method and equipment. In the method, a camera on the head-mounted display device is periodically exposed alternately with a first exposure time length and a second exposure time length, wherein the first exposure time length is longer than the second exposure time length. By sending a light frequency flashing signal and a synchronization signal to the hand-held handle, the shooting of the camera in the second exposure time length is strictly synchronized with the flickering of the LED light ring on the hand-held handle, so that the light spot image when the LED light ring is lighted in the second exposure time length is collected. In this way, the 6DOf poses of the two are quickly and accurately calculated by using the environmental image shot in the first exposure time length, the IMU data of the head-mounted display device itself and the light spot image shot in the second exposure time length and the IMU data of the hand-held handle, so that the quick and accurate control of the hand-held handle on the head-mounted display device is ensured, and the interactive response result and the use experience are improved.

Description

Technical Field

[0001] This application relates to the field of Virtual Reality (VR) / Augmented Reality (AR) technology, and provides a method and device for synchronous positioning of a head-mounted display device and a controller. Background Technology

[0002] With the development of VR and AR technologies, head-mounted displays (HMDs), such as VR glasses and AR glasses, are widely used in various industries, including education and training, fire drills, virtual driving, and real estate. While providing users with an immersive visual experience, the natural and smooth interaction methods are the most important factor in enhancing user satisfaction.

[0003] Currently, interaction methods for head-mounted display devices include head control and handheld control. Handheld control of the head-mounted display device is as crucial as a mouse control of a PC. Typically, a head-mounted display device has at least two cameras, and the handheld controllers have multiple LEDs that flash at a fixed frequency. The head-mounted display device uses images captured by the cameras of the LEDs on the handheld controllers when they are lit to track and locate the controllers for interaction. Therefore, if the cameras fail to capture images of the LEDs being lit, the handheld controllers will fail to locate, preventing synchronized positioning between the head-mounted display device and the handheld controllers, thus affecting the interaction results. Summary of the Invention

[0004] This application provides a method and apparatus for synchronizing a head-mounted display device and a controller, which improves the robustness of controller positioning.

[0005] On one hand, this application provides a handle positioning method applied to a head-mounted display device. The camera on the head-mounted display device is configured with a first exposure time and a second exposure time, which alternate periodically. The first exposure time is longer than the second exposure time. The head-mounted display device communicates with a handle, which is equipped with a flashing LED ring. The method includes:

[0006] At a first moment before the camera takes a picture with the first exposure time, a light strobe signal is generated and sent to the handle;

[0007] According to the exposure cycle occupied by the second exposure time, a synchronization signal is sent to the handle at a second time interval. The synchronization signal is received by the handle within a first time interval at the second time interval, so that the handle determines the second time interval from the generation of the light strobe signal to the lighting of the LED ring based on the time information of the received synchronization signal and the time information of the generated light strobe signal, and lights up the LED ring according to the second time interval.

[0008] According to the exposure cycle occupied by the first exposure duration, the camera acquires the environmental image, and determines its own first 6DOf pose based on the environmental image and the built-in IMU data. According to the exposure cycle occupied by the second exposure duration, the camera acquires the light spot image when the LED ring is lit, and determines the second 6DOf pose of the handle based on the light spot image, the handle's IMU data and the first 6DOf pose.

[0009] On the other hand, embodiments of this application provide a synchronous positioning method for a head-mounted display device and a handle, applied to the handle, which is equipped with a flashing LED ring. The handle communicates with the head-mounted display device, and the camera on the head-mounted display device is configured with a first exposure time and a second exposure time, which alternate periodically, with the first exposure time being longer than the second exposure time. The method includes:

[0010] Receive the light strobe signal generated by the head-mounted display device at a first moment before the camera takes a picture with the first exposure duration;

[0011] According to the exposure cycle occupied by the second exposure duration, the head-mounted display device receives a synchronization signal sent at a second time interval, and the time of receiving the synchronization signal is after a first time interval from the second time interval;

[0012] Based on the time information of the synchronization signal and the time information of the lamp flashing signal, a second time interval from the generation of the lamp flashing signal to the lighting of the LED ring is determined;

[0013] The LED ring is lit at the second time interval, so that the head-mounted display device determines its first 6DOf pose based on the environmental image captured by the camera during the exposure period occupied by the first exposure time and the built-in IMU data, and determines the second 6DOf pose of the handle based on the spot image of the LED ring when it is lit, the IMU data of the handle, and the first 6DOf pose captured by the camera during the exposure period occupied by the second exposure time.

[0014] On the other hand, this application provides a head-mounted display device. The external camera of the head-mounted display device is configured with a first exposure time and a second exposure time. The first exposure time and the second exposure time alternate periodically. The first exposure time is longer than the second exposure time. The head-mounted display device has a built-in processor, memory, display, IMU and communication interface. The communication interface, the IMU, the display, the memory and the processor are connected through a bus.

[0015] The head-mounted display device communicates with a handle equipped with a flashing LED ring and an IMU via the communication interface, so that the handle controls the application interface displayed on the display.

[0016] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0017] At a first moment before the camera takes a picture with the first exposure time, a light strobe signal is generated and sent to the handle;

[0018] According to the exposure cycle occupied by the second exposure time, a synchronization signal is sent to the handle at a second time interval. The synchronization signal is received by the handle within a first time interval at the second time interval, so that the handle determines the second time interval from the generation of the light strobe signal to the lighting of the LED ring based on the time information of the received synchronization signal and the time information of the generated light strobe signal, and lights up the LED ring according to the second time interval.

[0019] According to the exposure cycle occupied by the first exposure duration, the camera acquires the environmental image, and determines its own first 6DOf pose based on the environmental image and the built-in IMU data. According to the exposure cycle occupied by the second exposure duration, the camera acquires the light spot image when the LED ring is lit, and determines the second 6DOf pose of the handle based on the light spot image, the handle's IMU data and the first 6DOf pose.

[0020] Optionally, after acquiring the environmental images captured by the camera within the first exposure time, the processor further executes:

[0021] Anomaly detection is performed on the environmental image captured by the camera during the current first exposure time. The anomaly detection includes at least one of overexposure, offset, and deviation.

[0022] When at least one anomaly is detected, the first exposure duration in subsequent exposure cycles is adjusted by increasing or decreasing based on the detection results within the time range corresponding to the camera's frame rate.

[0023] Optionally, the second exposure duration is greater than or equal to the illumination duration of the LED.

[0024] Optionally, the number of handles is two.

[0025] On the other hand, this application embodiment provides a handle with an externally mounted flashing LED ring, and the handle has a built-in processor, memory, communication interface and IMU, and the IMU, the communication interface, the memory and the processor are connected via a bus;

[0026] The handle communicates with the head-mounted display device through the communication interface to control the application interface displayed on the display of the head-mounted display device. The camera on the head-mounted display device is set with a first exposure time and a second exposure time. The first exposure time and the second exposure time alternate periodically, and the first exposure time is longer than the second exposure time.

[0027] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0028] Receive the light strobe signal generated by the head-mounted display device at a first moment before the camera takes a picture with the first exposure duration;

[0029] According to the exposure cycle occupied by the second exposure duration, the head-mounted display device receives a synchronization signal sent at a second time interval, and the time of receiving the synchronization signal is after a first time interval from the second time interval;

[0030] Based on the time information of the synchronization signal and the time information of the lamp flashing signal, a second time interval from the generation of the lamp flashing signal to the lighting of the LED ring is determined;

[0031] The LED ring is lit at the second time interval, so that the head-mounted display device determines its first 6DOf pose based on the environmental image captured by the camera during the exposure period occupied by the first exposure time and the built-in IMU data, and determines the second 6DOf pose of the handle based on the spot image of the LED ring when it is lit, the IMU data of the handle, and the first 6DOf pose captured by the camera during the exposure period occupied by the second exposure time.

[0032] Optionally, the processor determines a second time interval from the generation of the light strobe signal to the lighting of the LED ring based on the time information of the synchronization signal and the time information of the light strobe signal. Specifically, the operation is as follows:

[0033] Calculate the signal time difference between the second moment when the synchronization signal is sent and the first moment when the lamp strobe signal is generated;

[0034] The difference between the exposure period, the signal time difference, and the first time interval is used as the second time interval from the generation of the lamp flicker signal to the lighting of the LED ring.

[0035] Optionally, the second exposure duration is greater than or equal to the illumination duration of the LED.

[0036] Optionally, when the second exposure duration is greater than the illumination duration of the LED, the illumination duration of the LED ring is located in the middle part of the second exposure duration, and the formula for the second time interval is expressed as:

[0037] Δt=T-(t2-t1)-ΔT-(S1-S2) / 2

[0038] Wherein, T represents the exposure duration, t2 represents the second moment when the synchronization signal is sent, t1 represents the first moment when the lamp strobe signal is generated, ΔT represents the first time interval from the transmission of the synchronization signal to its reception, S1 represents the second exposure duration, S2 represents the lighting duration of the LED ring, and Δt represents the second time interval from the generation of the lamp strobe signal to the lighting of the LED ring.

[0039] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the steps of a synchronization positioning method for a head-mounted display device and a handheld device provided in embodiments of this application.

[0040] The beneficial effects of the synchronous positioning method for the head-mounted display device and the handle provided in this application embodiment are as follows:

[0041] The head-mounted display device communicates with the handle, which is equipped with a flashing LED ring. The camera on the head-mounted display device is set with a first exposure time and a second exposure time, which alternate periodically. The first exposure time is longer than the second exposure time, and the camera's shooting and the flashing of the LED ring are synchronized during the second exposure time, so as to ensure that the camera can capture the image of the LED ring when it is lit during the second exposure time for positioning the handle. In the synchronization process, the head-mounted display first generates a strobe signal and sends it to the controller at the first moment before the camera takes a picture with the first exposure duration. This signals the controller to illuminate the LED ring. Then, the head-mounted display sends a synchronization signal to the controller at the second moment of the exposure cycle occupied by the second exposure duration. This synchronization signal is received by the controller after a first time interval. Based on the time information of the received synchronization signal and the time information of the generated strobe signal, the controller determines the second time interval from the generation of the strobe signal to the illumination of the LED ring. The controller then illuminates the LED ring according to the second time interval. This allows the camera to capture the spot image of the LED ring when it is illuminated during the second exposure duration. Combined with the environmental image and IMU data captured by the camera during the first exposure duration, the 6DOf pose of the head-mounted display and the controller can be accurately located. Thus, accurate interaction between the two can be achieved based on the 6DOf pose.

[0042] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the handle button structure provided in an embodiment of this application;

[0045] Figure 2 This is a schematic diagram illustrating the interaction between the handle and the head-mounted display device provided in an embodiment of this application.

[0046] Figure 3 This is a schematic diagram of the structure of the camera and handle LED light ring of the head-mounted display device provided in the embodiments of this application;

[0047] Figure 4This is a schematic diagram illustrating the asynchronous camera capture and LED light ring flashing provided in an embodiment of this application.

[0048] Figure 5 A diagram illustrating the overall architecture of the method for synchronizing the positioning of a head-mounted display device and a handheld device;

[0049] Figure 6 This is a schematic diagram illustrating how to set the first and second exposure times;

[0050] Figure 7 A flowchart of the method on the head-mounted display device side in the synchronous positioning method of the head-mounted display device and the handle;

[0051] Figure 8 This is a schematic diagram illustrating the synchronization of camera capture and LED ring flashing in an embodiment of this application.

[0052] Figure 9 This is a diagram illustrating the effect of adjusting the first exposure duration;

[0053] Figure 10 A flowchart of the method for synchronous positioning of a head-mounted display device and a handheld device on the handheld side;

[0054] Figure 11 This is a flowchart illustrating the calculation method for the second time interval from the generation of the lamp flicker signal to the lighting of the LED ring.

[0055] Figure 12 A synchronization diagram showing that the second exposure time is greater than the flashing time of the LED ring is provided in an embodiment of this application;

[0056] Figure 13 This is a schematic diagram illustrating the interaction process between a head-mounted display device and a controller;

[0057] Figure 14 A flowchart illustrating the interaction method between a head-mounted display device and a controller;

[0058] Figure 15 This is a schematic diagram of the structure of a head-mounted display device;

[0059] Figure 16 The structure of the handle provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0061] Currently, AR / VR and other head-mounted display devices use two types of controller positioning and tracking functions: 3DOF (attitude only) positioning and 6DOF (attitude and position included) positioning. Due to its higher positioning accuracy, 6DOF is currently the mainstream interactive control method.

[0062] like Figure 1 As shown, the controller includes multiple operation buttons such as trigger buttons, side grip buttons, home button, back button, joystick, and AB buttons, and can also emit rays. During interaction, the ray emitted by the controller points towards the application interface of the head-mounted display, forming a cursor on the application interface. In this way, the user can use the buttons on the controller to operate the controls at the focus of the ray cursor to achieve interactive functions.

[0063] Taking two handles as an example, such as Figure 2 The diagram illustrates the interaction between the controllers and the head-mounted display device. The left and right controllers may correspond to different distances and angles from the head-mounted display device. When the user needs to click the "Confirm" control on the application interface, they can select one controller for interaction. For example, by moving the ray cursor emitted by the left controller to the "Confirm" control on the application interface, and then pulling the trigger button on the left controller, the head-mounted display shows the application interface corresponding to the "Confirm" control, thus achieving interaction with the head-mounted display device. Users can switch between using the left and right controllers for interaction.

[0064] Typically, head-mounted display devices have at least two external cameras, and the handle has an external ring of multiple LEDs that flash at a fixed frequency, such as... Figure 3 As shown (not shown), the head-mounted display device has a built-in inertial measurement unit (IMU), and similarly, the handle also has a built-in IMU.

[0065] In interactive scenarios of VR / AR applications, when using the IMU of two controllers moving within a spatial range and flashing LEDs for real-time positioning and tracking, two problems need to be solved: first, to achieve simultaneous localization and mapping (SLAM) spatial motion positioning of the head-mounted display device; and second, to achieve simultaneous positioning and tracking of the two controllers.

[0066] To achieve both of these functions simultaneously, it is necessary to ensure that the camera on the head-mounted display device can capture the user's environment in real time, and to ensure that the camera's shooting and the flashing of the LED ring on the handle are strictly synchronized, so as to realize SLAM spatial motion positioning and dual-handle 6DoF fast and accurate positioning and tracking.

[0067] However, since the head-mounted display and the left and right handles are independent of each other, when positioning is based on images captured by the camera, if the camera's shooting frequency on the head-mounted display is not synchronized with the flashing frequency of the LED rings on the dual handles, such as... Figure 4 As shown, within the exposure period T, the camera may not be able to capture the image when the LED ring is lit, which will cause the handle positioning to fail. It will be impossible to simultaneously achieve SLAM spatial motion positioning of the head-mounted display device and 6DoF positioning tracking of the dual handles, thus producing incorrect exchange results.

[0068] In view of this, this application provides a synchronous positioning method for a head-mounted display device and a controller. This method adjusts the camera's shooting mode, using a combination of long and short exposure times. The images captured during the long exposure time are used for SLAM spatial motion positioning of the head-mounted display device, while the short exposure time is strictly synchronized with the flashing duration of the LED rings on the dual controllers, ensuring that the camera can capture images of the LED rings on the dual controllers when they are lit. This achieves SLAM spatial motion positioning of the head-mounted display device and 6DoF fast positioning and tracking of the dual controllers. At the same time, this method also adjusts the long exposure time of the camera according to the actual environment, thereby improving positioning accuracy, which in turn improves the speed and accuracy of interactive response, resulting in improved user experience satisfaction.

[0069] See Figure 5 This diagram illustrates the overall architecture of a synchronized positioning method for a head-mounted display (HUD) and controllers. The controllers are equipped with LED rings, a power supply, and physical function keys. The controller's microcontroller unit (MCU) controls the LED rings to blink at a fixed frequency via a reset circuit. The controllers and the HUD communicate via their respective wireless modules. The HUD uses environmental images captured by its camera and its own IMU data to calculate its 6DoF pose using a SLAM algorithm. For real-time positioning and tracking of both controllers, the camera on the HUD and the LED rings on the controllers must be synchronized; that is, the LED rings on the controllers must blink regularly. The HUD calculates the 6DoF pose of the controllers relative to the HUD based on the light spot images captured by the camera when the LED rings on the controllers are lit, combined with the controller's IMU data and the HUD's 6DoF pose. This enables SLAM spatial motion positioning of the HUD and 6DoF positioning and tracking of the controllers, ultimately allowing the controllers to interactively control the HUD.

[0070] In the embodiments of this application, in order to achieve synchronous positioning of the head-mounted display device and the handle, the camera's exposure period T is set to include different exposure durations, denoted as the first exposure duration and the second exposure duration, respectively. The first exposure duration and the second exposure duration alternate periodically, with the first exposure duration being longer than the second exposure duration. That is, the bit width of the first exposure duration within the exposure period is greater than the bit width of the second exposure duration within the exposure period. Figure 6 As shown. The images captured by the camera during the first exposure time are used for SLAM spatial motion positioning of the head-mounted display device. The start time of the second exposure time is strictly synchronized with the lighting time of the LED ring on the dual handles. Thus, the light spot images of the LED ring captured by the camera during the second exposure time can be used for 6DoF positioning and tracking of the dual handles. The camera's exposure period is determined based on the international shooting frame rate.

[0071] Since the left and right controllers are independent of the head-mounted display device, the subsequent description of the synchronization positioning method between the head-mounted display device and the controllers will not distinguish between the left and right controllers. The process on the head-mounted display device side in this method is as follows: Figure 7 As shown, it mainly includes the following steps:

[0072] S701: At the very first moment before the camera takes a picture with the first exposure time, a light strobe signal is generated and sent to the handle.

[0073] Due to the limited space in the handle, the power supply of the handle is usually small. In order to reduce the power consumption of the handle, the LED light ring is usually lit intermittently at a certain frequency, that is, it flashes. Therefore, in order to ensure the synchronous positioning of the head-mounted display device and the handle, a light strobe signal is generated and sent to the handle at the first moment before the camera of the head-mounted display device takes a picture with the first exposure time, so as to notify the handle to light up the LED light ring.

[0074] S702: Sends a synchronization signal to the handle at regular intervals according to the exposure cycle occupied by the second exposure duration.

[0075] To ensure 6DOf positioning and tracking of the handheld device, the camera's second exposure duration needs to be synchronized with the LED ring's flashing. Therefore, the head-mounted display sends a synchronization signal (Sync) to the handheld device at regular intervals during the second exposure. The timing of this synchronization signal transmission is the same as the exposure period occupied by the second exposure duration, and it is received by the handheld device after a first time interval, meaning the transmission duration of the synchronization signal is the first time interval. Based on the timing information of the received synchronization signal and the generated LED flash signal, the handheld device can determine the second time interval from the generation of the flash signal to the LED ring illuminating, and illuminate the LED ring according to this second time interval. This ensures that the camera can capture the light spot image of the LED ring illuminating each time within the second exposure duration.

[0076] The calculation process for the second time interval is described in the handle side description and will not be repeated here.

[0077] Optionally, the second moment of sending the Sync signal is the start moment of the first exposure duration.

[0078] like Figure 8 The diagram shows the result of synchronizing the camera's second exposure time with the flashing of the LED ring. At the beginning of the second exposure time, the LED ring starts to light up, thus ensuring that the camera can capture the light spot image when the LED ring is lit during the second exposure.

[0079] S703: According to the exposure cycle occupied by the first exposure duration, acquire the environmental image captured by the camera, and determine its own first 6DOf pose based on the environmental image and the built-in IMU data. Also, according to the exposure cycle occupied by the second exposure duration, acquire the light spot image of the LED ring when it is lit at the second time interval captured by the camera, and determine the second 6DOf pose of the handle based on the light spot image, the IMU data of the handle, and the first 6DOf pose.

[0080] Since head-mounted displays have built-in IMUs, they can acquire the three-axis acceleration and three-axis angular velocity of the head-mounted display. By pre-integrating the data acquired by the IMU, the 6DoF pose of the head-mounted display can be obtained in real time. However, the IMU's estimated pose has accumulated errors, which increase with the length of time. Therefore, the current pose of the head-mounted display can be calculated using computer vision algorithms from the environmental images captured by the camera within the first exposure time. The estimated 6DoF pose can then be optimized using the current pose to obtain the final first 6DoF pose of the head-mounted display. Similarly, the controller also has a built-in IMU, which can collect the controller's three-axis acceleration and three-axis angular velocity, and transmit the data collected by the IMU to the head-mounted display device. The head-mounted display device performs prediction and segmentation on the controller's IMU data, and calculates the controller's second 6DoF pose based on the light spot image of the LED ring when it is lit, which is captured by the camera during the second exposure time. Combined with its own 6DoF pose, the controller's 6DoF pose is also unified to the camera coordinate system, thereby achieving synchronous positioning of the controller and the head-mounted display device.

[0081] In one embodiment, considering the changes in the camera acquisition environment, the first exposure time of the camera can be adjusted during the positioning process. That is, the first exposure time may be the same or different in different exposure cycles, thereby reducing the impact of environmental changes on feature points in the image and improving the overall positioning accuracy.

[0082] In practice, anomaly detection is performed on the environmental image captured by the camera within the current first exposure duration. This detection includes, but is not limited to, checking for overexposure, whether the offset from the previous frame's environmental image exceeds a preset threshold, and whether there are any deviations in the current environment. If at least one of these issues is detected, the first exposure duration in subsequent exposure cycles is adjusted by increasing or decreasing the duration based on the detection results, within the time range corresponding to the camera's frame rate.

[0083] For example, assuming the camera's frame rate is 25 Hz, meaning the camera's exposure cycle T = 4 milliseconds (ms), the duration of the first exposure is in the range of 2–4 ms. If the first exposure duration of the current environment image is 2 ms, and underexposure is detected, the first exposure duration in the next exposure cycle can be adjusted to 3 ms. Figure 9 As shown.

[0084] It should be noted that the embodiments of this application do not impose restrictive requirements on the anomaly detection algorithm for environmental images. Deep learning algorithms based on neural networks or traditional image processing algorithms can be used.

[0085] In the synchronous positioning method of the head-mounted display device and the controller in the embodiments of this application, the process on the controller side is as follows: Figure 10 As shown, it mainly includes the following steps:

[0086] S1001: Receives the light strobe signal generated by the head-mounted display device at the first moment before the camera takes a picture with the first exposure time.

[0087] Since the LED ring on the handle is illuminated in a flashing manner, the head-mounted display generates a strobe signal and sends it to the handle immediately before the camera takes a picture at the first exposure time. The handle then illuminates the LED ring based on the strobe signal. However, because the flashing time of the LED ring may not be synchronized with the second exposure time of the camera, the flashing frequency of the LED ring may differ from the exposure frequency of the camera. Therefore, it is necessary to synchronize the second exposure time of the camera with the flashing of the LED ring. This requires calculating the synchronization time interval, which is the time interval after the head-mounted display generates the strobe signal and the handle's LED ring illuminates. In this way, the light spot image of the LED ring is captured within the second exposure time of the camera, thereby locating the 6DOf pose of the handle.

[0088] S1002: Receives the synchronization signal sent by the head-mounted display device at a second time interval according to the exposure cycle occupied by the second exposure duration.

[0089] During the exposure cycle, when the camera acquires images at the second exposure duration, to ensure that the image of the LED light spot is captured when it is bright, the handle periodically receives synchronization signals from the head-mounted display to control the LED light ring to synchronize with the camera's second exposure duration. The synchronization result is as follows: Figure 8 As shown. The synchronization signal is sent by the head-mounted display device at the second moment and received by the handle after the first time interval. The timing of the synchronization signal transmission period is the exposure period occupied by the second exposure time, thereby ensuring that the camera can capture the light spot image when the LED light ring is lit every time it exposes for the second time.

[0090] S1003: Determine the second time interval from the generation of the lamp strobe signal to the lighting of the LED ring based on the time information of the synchronization signal and the time information of the lamp strobe signal.

[0091] In practice, the calculation process for the second time interval is as follows: Figure 11 As shown, it mainly includes the following steps:

[0092] S1003_1: Calculate the signal time difference between the second moment of sending the synchronization signal and the first moment of generating the lamp flashing signal.

[0093] S1003_2: The difference between the exposure period, the signal time difference, and the first time interval is used as the second time interval from the generation of the lamp flicker signal to the lighting of the LED ring. The formula for the second time interval is as follows:

[0094] Δt=T-(t2-t1)-ΔT Formula 1

[0095] Where T represents the exposure time, t2 represents the second moment of sending the synchronization signal, t1 represents the first moment of generating the lamp strobe signal, ΔT represents the first time interval from sending to receiving the synchronization signal, and Δt represents the second time interval from generating the lamp strobe signal to lighting up the LED ring.

[0096] S1004: The LED ring is lit up at a second time interval during the second exposure duration, so that the head-mounted display device can determine its first 6DOf pose based on the environmental image and built-in IMU data acquired by the camera during the exposure cycle occupied by the first exposure duration, and determine the second 6DOf pose of the handle based on the spot image of the LED ring when it is lit, the IMU data of the handle, and the first 6DOf pose acquired by the camera during the exposure cycle occupied by the second exposure duration.

[0097] The pose calculation process is described in the description of head-mounted display device measurement, and will not be repeated here.

[0098] In one embodiment, in order to reduce the amount of computation on the handle side, the calculation of the second time interval can be performed by the head-mounted display device, and the handle can be notified after the second time interval is calculated, so that the handle can light up the LED ring according to the second time interval.

[0099] In the embodiments of this application, in order to ensure the synchronous shooting effect of the camera and the LED light ring, the second exposure time of the camera is greater than or equal to the lighting time of the LED light.

[0100] In one example, when the camera's second exposure duration is longer than the LED's illumination duration, the LED ring's illumination duration falls within the middle of the camera's second exposure duration, such as... Figure 12 As shown, the formula for the second time interval is as follows:

[0101] Δt=T-(t2-t1)-ΔT-(S1-S2) / 2 Formula 2

[0102] S1 represents the second exposure time of the camera, and S2 represents the illumination time of the LED ring.

[0103] For example, assuming the camera's second exposure time is 500us and the LED ring's illumination time is 100us, using Formula 2, we can see that the LED ring starts flashing after the camera has been exposed for 200us within the second exposure time.

[0104] In the synchronization positioning method for a head-mounted display and a controller provided in this application embodiment, the interaction process between the head-mounted display device and the controller is as follows: Figure 13 As shown, firstly, the head-mounted display device sends a strobe signal to each of the two hands at the first moment before the camera takes a picture with the first exposure time to notify the two hands to light up the LED ring. Then, at the second moment within the exposure cycle occupied by the second exposure time, the head-mounted display device sends a synchronization signal to the two hands at regular intervals. This allows the two hands to calculate the time interval from when the head-mounted display device generates the strobe signal to when the LED ring is lit, based on the Sync time information and the strobe time information. This keeps the flashing of the LED ring synchronized with the second short exposure time, thereby achieving synchronized positioning of the head-mounted display device and the two hands.

[0105] The interaction process between head-mounted display devices and controllers is as follows: Figure 14 As shown, it mainly includes the following steps:

[0106] S1401: The head-mounted display device generates a lamp strobe signal at the first moment before the exposure cycle occupied by the first exposure time.

[0107] S1402: The head-mounted display device sends an equal-frequency flicker signal to the dual hands.

[0108] S1403: The head-mounted display device sends a synchronization signal to each of the two handles at the second moment of the exposure cycle occupied by the second exposure time.

[0109] S1404: After the first time interval, both handsets receive a synchronization signal.

[0110] S1405: The two handles respectively calculate the signal time difference between the second moment of sending the synchronization signal and the first moment of generating the light flashing signal.

[0111] S1406: The difference between the exposure period, the signal time difference, and the first time interval is used as the second time interval from the generation of the lamp strobe signal to the lighting of the LED ring.

[0112] S1407: The two handles light up their respective LED rings according to the second time interval.

[0113] S1408: The head-mounted display device uses a camera to acquire environmental images during the first exposure time, as well as its own IMU data, to calculate its first 6DoF pose.

[0114] S1409: The head-mounted display device uses the light spot images of the LED rings of the two handles when they are lit at the second time interval, which are captured by the camera during the second exposure time, as well as the acquired IMU data of the handles and their own first 6DoF pose, to calculate the second 6DoF pose of each of the two handles.

[0115] In the synchronous positioning method for a head-mounted display device and a handle provided in this application embodiment, different exposure durations are set within the camera's exposure cycle. A strobe signal is sent to the handle at a first moment before the exposure cycle occupied by the first exposure duration, and a synchronization signal is sent to the handle at a second moment within the exposure cycle occupied by the second exposure duration. Based on the time difference between these two signals, combined with the transmission duration of the synchronization signal (the first time interval) and the exposure cycle, the handle calculates the second time interval from the generation of the strobe signal to the lighting of the LED ring. This achieves strict synchronization between the camera's second exposure duration and the flashing of the handle's LED ring, ensuring both real-time acquisition of environmental information by the camera and strict synchronization between the camera's shooting and the flashing of the handle's LED ring. This enables SLAM spatial positioning of the head-mounted display device and fast, accurate 6DoF positioning and tracking by the dual handles, thereby ensuring rapid and accurate control of the head-mounted display device by the dual handles and improving interactive response results and user experience.

[0116] Based on the same technical concept, this application provides a head-mounted display device, which can be a virtual reality device or an augmented reality device. The head-mounted display device can implement the steps of the above-mentioned synchronous positioning method of a head-mounted display device and a controller, and can achieve the same technical effect.

[0117] See Figure 15 The head-mounted display device includes an external camera 1501 with a first exposure time and a second exposure time, which alternate periodically. The first exposure time is longer than the second exposure time. The head-mounted display device has a built-in processor 1502, a memory 1503, a display 1504, an IMU 1505, and a communication interface 1506. The camera 1501, the communication interface 1506, the IMU 1505, the display 1504, the memory 1503, and the processor 1502 are connected via a bus 1507.

[0118] The head-mounted display device communicates with a handle equipped with a flashing LED ring and an IMU via the communication interface 1506, so that the handle controls the application interface displayed on the display 1504.

[0119] The memory 1503 stores a computer program, and the processor 1502 performs the following operations according to the computer program:

[0120] At a first moment before the camera 1501 takes a picture with the first exposure duration, a light strobe signal is generated and sent to the handle;

[0121] According to the exposure cycle occupied by the second exposure time, a synchronization signal is sent to the handle at a second time interval. The synchronization signal is received by the handle within a first time interval at the second time interval, so that the handle determines the second time interval from the generation of the light strobe signal to the lighting of the LED ring based on the time information of the received synchronization signal and the time information of the generated light strobe signal, and lights up the LED ring according to the second time interval.

[0122] According to the exposure cycle occupied by the first exposure duration, the environmental image captured by the camera 1501 is acquired, and the first 6DOf pose of the device is determined based on the environmental image and the built-in IMU data. According to the exposure cycle occupied by the second exposure duration, the light spot image of the LED ring when it is lit is acquired by the camera 1501, and the second 6DOf pose of the device is determined based on the light spot image, the IMU data of the handle, and the first 6DOf pose.

[0123] Optionally, after acquiring the environmental images captured by the camera within the first exposure time, the processor 1502 further executes:

[0124] Anomaly detection is performed on the environmental image captured by the camera during the current first exposure time. The anomaly detection includes at least one of overexposure, offset, and deviation.

[0125] When at least one anomaly is detected, the first exposure duration in the subsequent exposure cycle is adjusted by increasing or decreasing according to the detection result within the time range corresponding to the acquisition frame rate of the camera 1501.

[0126] Optionally, the second exposure duration is greater than or equal to the illumination duration of the LED.

[0127] Optionally, the number of handles is two.

[0128] It should be noted that, Figure 15 This is merely an example illustrating the hardware necessary for a head-mounted display device to perform the synchronization positioning method steps of a head-mounted display device and a handle as provided in the embodiments of this application. Not shown, the head-mounted display device may also include conventional hardware such as speakers, microphones, left and right spectacle lenses, and a power supply.

[0129] Based on the same technical concept, this application provides a handle that can implement the steps of the above-described synchronous positioning method for a head-mounted display device and a handle, and achieve the same technical effect.

[0130] See Figure 16 The handle has an external flashing LED ring 1601, and the handle has a built-in processor 1602, memory 1603, communication interface 1604 and IMU 1605. The LED ring 1601, the IMU 1605, the communication interface 1604, the memory 1603 and the processor 1602 are connected via a bus 1606.

[0131] The handle communicates with the head-mounted display device through the communication interface 1604 to control the application interface displayed on the display of the head-mounted display device. The camera on the head-mounted display device is set with a first exposure time and a second exposure time. The first exposure time and the second exposure time alternate periodically, and the first exposure time is longer than the second exposure time.

[0132] The memory 1603 stores a computer program, and the processor 1602 performs the following operations according to the computer program:

[0133] Receive the light strobe signal generated by the head-mounted display device at a first moment before the camera takes a picture with the first exposure duration;

[0134] According to the exposure cycle occupied by the second exposure duration, the head-mounted display device receives a synchronization signal sent at a second time interval, and the time of receiving the synchronization signal is after a first time interval from the second time interval;

[0135] Based on the time information of the synchronization signal and the time information of the lamp flashing signal, a second time interval from the generation of the lamp flashing signal to the lighting of the LED ring is determined;

[0136] The LED ring 1601 is lit according to the second time interval, so that the head-mounted display device determines its first 6DOf pose based on the environmental image and built-in IMU data acquired by the camera during the exposure period occupied by the first exposure time, and determines the second 6DOf pose of the handle based on the spot image of the LED ring when it is lit, the IMU data of the handle, and the first 6DOf pose acquired by the camera during the exposure period occupied by the second exposure time.

[0137] Optionally, the processor 1602 determines a second time interval from the generation of the light strobe signal to the lighting of the LED ring based on the time information of the synchronization signal and the time information of the light strobe signal. Specifically, the operation is as follows:

[0138] Calculate the signal time difference between the second moment when the synchronization signal is sent and the first moment when the lamp strobe signal is generated;

[0139] The difference between the exposure period, the signal time difference, and the first time interval is used as the second time interval from the generation of the lamp flicker signal to the lighting of the LED ring.

[0140] Optionally, the second exposure duration is greater than or equal to the illumination duration of the LED.

[0141] Optionally, when the second exposure duration is greater than the illumination duration of the LED, the illumination duration of the LED ring is located in the middle part of the second exposure duration, and the formula for the second time interval is expressed as:

[0142] Δt=T-(t2-t1)-ΔT-(S1-S2) / 2

[0143] Wherein, T represents the exposure duration, t2 represents the second moment when the synchronization signal is sent, t1 represents the first moment when the lamp strobe signal is generated, ΔT represents the first time interval from the transmission of the synchronization signal to its reception, S1 represents the second exposure duration, S2 represents the lighting duration of the LED ring, and Δt represents the second time interval from the generation of the lamp strobe signal to the lighting of the LED ring.

[0144] in, Figure 15 and Figure 16 The memory in the memory can be volatile memory, such as random-access memory (RAM); it can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it can be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory can be a combination of the above-mentioned types of memory.

[0145] A processor may include one or more central processing units (CPUs) or digital processing units, etc.

[0146] It should be noted that, Figure 16 This is merely an example illustrating the hardware necessary for the handle to perform the synchronization positioning method steps of a head-mounted display device and a handle as provided in the embodiments of this application. Not shown, the handle may also include conventional hardware such as physical buttons and a power supply.

[0147] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform a synchronous positioning method for a head-mounted display device and a handle as described in the foregoing embodiments.

[0148] This application also provides a computer program product for storing a computer program for executing a synchronous positioning method for a head-mounted display device and a handheld device as described in the foregoing embodiments.

[0149] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of synchronized positioning of a head-mounted display and a hand grip, characterized in that, An application is made to a head-mounted display device, wherein a camera on the head-mounted display device is configured with a first exposure time and a second exposure time, the first exposure time and the second exposure time alternating periodically, the first exposure time being longer than the second exposure time, the head-mounted display device communicating with a handle, the handle being equipped with a flashing LED ring, the method comprising: At a first moment before the camera takes a picture with the first exposure time, a light strobe signal is generated and sent to the handle; According to the exposure cycle occupied by the second exposure duration, a synchronization signal is sent to the handle at a second time interval. The synchronization signal is sent from the second time interval to the first time interval after which it is received by the handle. The handle determines the second time interval from the generation of the light strobe signal to the lighting of the LED ring based on the time information of the received synchronization signal and the time information of the generated light strobe signal, combined with the exposure cycle and the first time interval, and lights up the LED ring according to the second time interval. The second exposure duration is synchronized with the lighting event of the LED ring, so that the camera can capture the light spot image of the LED ring when it is lit within the second exposure duration.

2. The method as described in claim 1, characterized in that, The second time interval was obtained in the following manner: Calculate the signal time difference between the second moment when the synchronization signal is sent and the first moment when the lamp strobe signal is generated; The difference between the exposure period, the signal time difference, and the first time interval is used as the second time interval.

3. The method as described in claim 2, characterized in that, When the second exposure duration is greater than the illumination duration of the LED ring, the illumination duration of the LED ring is located in the middle part of the second exposure duration. The second time interval is calculated using the following formula: Wherein, T represents the exposure period, t2 represents the second moment, t1 represents the first moment, ΔT represents the first time interval, S1 represents the second exposure duration, S2 represents the illumination duration of the LED ring, and Δt represents the second time interval.

4. The method of claim 1, wherein, The method further includes: Anomaly detection is performed on the environmental image captured by the camera during the current first exposure time. The anomaly detection includes at least one of overexposure, offset, and deviation. When at least one anomaly is detected, the first exposure duration in subsequent exposure cycles is adjusted by increasing or decreasing based on the detection results within the time range corresponding to the camera's frame rate.

5. A method for synchronously positioning a head-mounted display device and a handle, characterized in that, An application is made to a handle, the handle being equipped with a flashing LED ring, the handle communicating with a head-mounted display device, the camera on the head-mounted display device being configured with a first exposure time and a second exposure time, the first exposure time and the second exposure time alternating periodically, the first exposure time being longer than the second exposure time, the method comprising: Receive the light strobe signal generated by the head-mounted display device at a first moment before the camera takes a picture with the first exposure duration; According to the exposure cycle occupied by the second exposure duration, the head-mounted display device receives a synchronization signal sent at a second time interval, and the time of receiving the synchronization signal is after a first time interval from the second time interval; Based on the time information of the synchronization signal and the time information of the lamp strobe signal, and in combination with the exposure period and the first time interval, a second time interval from the generation of the lamp strobe signal to the lighting of the LED ring is determined; The LED ring is lit up at the second time interval.

6. A head-mounted display device, characterized in that, The external camera of the head-mounted display device is configured with a first exposure time and a second exposure time, which alternate periodically. The first exposure time is longer than the second exposure time. The head-mounted display device has a built-in processor, memory, display, IMU and communication interface. The communication interface, the IMU, the display, the memory and the processor are connected through a bus. The head-mounted display device communicates with a handle equipped with a flashing LED ring and an IMU via the communication interface, so that the handle controls the application interface displayed on the display. The memory stores a computer program, and the processor performs the following operations according to the computer program: At a first moment before the camera takes a picture with the first exposure time, a light strobe signal is generated and sent to the handle; According to the exposure cycle occupied by the second exposure duration, a synchronization signal is sent to the handle at a second time interval. The synchronization signal is sent from the second time interval to the first time interval after which it is received by the handle. The handle determines the second time interval from the generation of the light strobe signal to the lighting of the LED ring based on the time information of the received synchronization signal and the time information of the generated light strobe signal, combined with the exposure cycle and the first time interval, and lights up the LED ring according to the second time interval. The second exposure duration is synchronized with the lighting event of the LED ring, so that the camera can capture the light spot image of the LED ring when it is lit within the second exposure duration.

7. A handle, characterized in that, The handle has an external flashing LED light ring, and the handle has a built-in processor, memory, communication interface and IMU. The IMU, the communication interface, the memory and the processor are connected via a bus. The handle communicates with the head-mounted display device through the communication interface to control the application interface displayed on the head-mounted display device's screen. The camera on the head-mounted display device is set with a first exposure time and a second exposure time. The first exposure time and the second exposure time alternate periodically. The first exposure time is longer than the second exposure time, and the second exposure time is longer than or equal to the lighting time of the LED light ring. The memory stores a computer program, and the processor performs the following operations according to the computer program: Receive the light strobe signal generated by the head-mounted display device at a first moment before the camera takes a picture with the first exposure duration; According to the exposure cycle occupied by the second exposure duration, the head-mounted display device receives a synchronization signal sent at a second time interval, and the time of receiving the synchronization signal is after a first time interval from the second time interval; Based on the time information of the synchronization signal and the time information of the lamp strobe signal, and in combination with the exposure period and the first time interval, a second time interval from the generation of the lamp strobe signal to the lighting of the LED ring is determined; The LED ring is lit up at the second time interval.

Images