Synchronous positioning method and device for head-mounted display device and handle

Abstract

The invention relates to the technical field of VR / AR, and provides a synchronous positioning method and device for a head-mounted display device and a handle. According to the method, a camera on the head-mounted display device periodically and alternately exposes according to a first exposure time length and a second exposure time length, the first exposure time length is larger than the second exposure time length, a lamp stroboscopic signal and a synchronizing signal are sent to a handle, shooting of the camera within the second exposure time length and flickering of an LED lamp ring on the handle are strictly synchronized, and therefore the shooting accuracy of the head-mounted display device is improved. Therefore, a light spot image when the LED lamp ring is lightened is acquired in the second exposure time, so that the 6DOF pose of the environment image and the IMU data of the head-mounted display equipment in the first exposure time as well as the light spot image and the IMU data of the handle shot in the second exposure time is quickly and accurately calculated, and the accuracy of the 6DOF pose of the environment image and the IMU data of the head-mounted display equipment is improved. Therefore, rapid and accurate control of the handle on the head-mounted display device is ensured, and the interaction response result and the use experience degree are improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) / augmented reality (AR) technology, and provides a method and device for synchronous positioning of a head-mounted display device and a handle. Background Art

[0002] With the development of VR and AR technologies, head mounted display devices (HMD) such as VR glasses and AR glasses have been widely used in various industries, such as education and training, fire drills, virtual driving, real estate, etc. Head mounted display devices are used in various fields. While head mounted display devices bring users an immersive visual experience, the natural and smooth interaction method has become the most important embodiment of improving user experience satisfaction.

[0003] At present, the interaction modes of head-mounted display devices include head control and handle control, among which the handle is as important to the head-mounted display device as the mouse is to the PC. Generally, at least two cameras are provided on the head-mounted display device, and a plurality of LED lights flashing at a fixed frequency are provided on the handle. The head-mounted display device uses the images taken by the camera when the LED lights on the handle are lit to locate and track the handle to achieve interaction between the two. In this way, when the camera fails to capture the image when the LED lights are lit, the handle positioning will fail, and the synchronous positioning of the head-mounted display device and the handle cannot be completed, thereby affecting the interaction result. Summary of the invention

[0004] The embodiments of the present application provide a method and device for synchronizing a head-mounted display device and a handle, so as to improve the robustness of handle positioning.

[0005] On the one hand, an embodiment of the present application provides a handle positioning method, which is applied to a head-mounted display device, wherein a camera on the head-mounted display device is provided with a first exposure time and a second exposure time, the first exposure time and the second exposure time are periodically alternated, the first exposure time is greater than the second exposure time, the head-mounted display device communicates with a handle, and the handle is provided with a flashing LED light ring, the method comprising:

[0006] At a first moment before the camera takes a photo with the first exposure duration, a light strobe signal is generated and sent to the handle;

[0007] According to the exposure cycle occupied by the second exposure duration, a synchronization signal is periodically sent to the handle at a second moment, and the synchronization signal is received by the handle within a first time interval at the second moment, so that the handle determines a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to time information of the received synchronization signal and time information of the generated light strobe signal, and lights up the LED light ring according to the second time interval;

[0008] According to the exposure period occupied by the first exposure time, the environment image captured by the camera is obtained, and according to the environment image and the built-in IMU data, the first 6DOf posture of the handle is determined. In addition, according to the exposure period occupied by the second exposure time, the light spot image when the LED light ring is lit is obtained, and the second 6DOf posture of the handle is determined according to the light spot image, the IMU data of the handle and the first 6DOf posture.

[0009] On the other hand, an embodiment of the present application provides a synchronous positioning method for a head-mounted display device and a handle, which is applied to the handle, the handle is provided with a flashing LED light ring, the handle communicates with the head-mounted display device, and a camera on the head-mounted display device is provided with a first exposure time and a second exposure time, the first exposure time and the second exposure time are periodically alternated, and the first exposure time is greater than the second exposure time, and the method includes:

[0010] receiving a light strobe signal generated by the head mounted display device at a first moment before the camera takes a picture with the first exposure duration;

[0011] receiving, according to the exposure period occupied by the second exposure duration, a synchronization signal regularly sent by the head mounted display device at a second moment, wherein a first time interval has elapsed from a time when the synchronization signal is received to a time when the second moment is received;

[0012] Determine a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to the time information of the synchronization signal and the time information of the light strobe signal;

[0013] Light up the LED light ring at the second time interval so that the head-mounted display device determines its own first 6DOf posture according to the environmental image collected by the camera during the exposure period occupied by the first exposure time and the built-in IMU data, and determines the second 6DOf posture of the handle according to the light spot image when the LED light ring is lit, collected by the camera during the exposure period occupied by the second exposure time, the IMU data of the handle, and the first 6Dof posture.

[0014] On the other hand, an embodiment of the present application provides a head-mounted display device, wherein an external camera of the head-mounted display device is provided with a first exposure duration and a second exposure duration, wherein the first exposure duration and the second exposure duration are periodically alternated, and the first exposure duration is greater than the second exposure duration, and the head-mounted display device is built-in with a processor, a memory, a display, an IMU, and a communication interface, wherein the communication interface, the IMU, the display, the memory, and the processor are connected via a bus;

[0015] The head mounted display device communicates with a handle provided with a flashing LED light ring and an IMU through the communication interface, so that the handle controls the application interface displayed on the display;

[0016] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0017] At a first moment before the camera takes a photo with the first exposure duration, a light strobe signal is generated and sent to the handle;

[0018] According to the exposure cycle occupied by the second exposure duration, a synchronization signal is periodically sent to the handle at a second moment, and the synchronization signal is received by the handle within a first time interval at the second moment, so that the handle determines a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to time information of the received synchronization signal and time information of the generated light strobe signal, and lights up the LED light ring according to the second time interval;

[0019] According to the exposure period occupied by the first exposure time, the environment image captured by the camera is obtained, and according to the environment image and the built-in IMU data, the first 6DOf posture of the handle is determined. In addition, according to the exposure period occupied by the second exposure time, the light spot image when the LED light ring is lit is obtained, and the second 6DOf posture of the handle is determined according to the light spot image, the IMU data of the handle and the first 6DOf posture.

[0020] Optionally, after acquiring the environment image captured by the camera within the first exposure duration, the processor further executes:

[0021] Performing anomaly detection on the environment image acquired by the camera during the current first exposure duration, wherein the anomaly detection includes at least one of overexposure, offset, and deviation;

[0022] When at least one abnormality is detected, within the time range corresponding to the acquisition frame rate of the camera, the first exposure time in the subsequent exposure cycle is adjusted to increase or decrease according to the detection result.

[0023] Optionally, the second exposure duration is greater than or equal to the lighting duration of the LED light.

[0024] Optionally, there are two handles.

[0025] On the other hand, an embodiment of the present application provides a handle, wherein the handle is provided with a flashing LED light ring, and the handle is provided with a processor, a memory, a communication interface and an IMU, wherein the IMU, the communication interface, the memory and the processor are connected via a bus;

[0026] The handle communicates with the head-mounted display device through the communication interface to control the application interface displayed on the display of the head-mounted display device, wherein the camera on the head-mounted display device is set with a first exposure time and a second exposure time, the first exposure time and the second exposure time are periodically alternated, and the first exposure time is greater than the second exposure time;

[0027] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0028] receiving a light strobe signal generated by the head mounted display device at a first moment before the camera takes a picture with the first exposure duration;

[0029] receiving, according to the exposure period occupied by the second exposure duration, a synchronization signal regularly sent by the head mounted display device at a second moment, wherein a first time interval has elapsed from a time when the synchronization signal is received to a time when the second moment is received;

[0030] Determine a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to the time information of the synchronization signal and the time information of the light strobe signal;

[0031] Light up the LED light ring at the second time interval so that the head-mounted display device determines its own first 6DOf posture according to the environmental image collected by the camera during the exposure period occupied by the first exposure time and the built-in IMU data, and determines the second 6DOf posture of the handle according to the light spot image when the LED light ring is lit, collected by the camera during the exposure period occupied by the second exposure time, the IMU data of the handle, and the first 6Dof posture.

[0032] Optionally, the processor determines a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to the time information of the synchronization signal and the time information of the light strobe signal, and the specific operation is:

[0033] Calculate the signal time difference between the second time of sending the synchronization signal and the first time of generating the light strobe signal;

[0034] The difference between the exposure period, the signal time difference and the first time interval is used as the second time interval from the generation of the light strobe signal to the lighting of the LED light ring.

[0035] Optionally, the second exposure duration is greater than or equal to the lighting duration of the LED light.

[0036] Optionally, when the second exposure duration is greater than the lighting duration of the LED light, the lighting duration of the LED light ring is located in the middle part of the second exposure duration, and the formula for the second time interval is expressed as:

[0037] Δt=T-(t2-t1)-ΔT-(S1-S2) / 2

[0038] Wherein, T represents the exposure duration, t2 represents the second moment of sending the synchronization signal, t1 represents the first moment of generating the light strobe signal, ΔT represents the first time interval from sending to receiving the synchronization signal, S1 represents the second exposure duration, S2 represents the lighting duration of the LED light ring, and Δt represents the second time interval from the generation of the light strobe signal to the lighting of the LED light ring.

[0039] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a method for synchronous positioning of a head-mounted display device and a handle provided in an embodiment of the present application.

[0040] The beneficial effects of the synchronous positioning method of the head mounted display device and the handle provided in the embodiment of the present application are as follows:

[0041] The head-mounted display device communicates with a handle, a flashing LED light ring is provided on the handle, and a camera on the head-mounted display device is provided with a first exposure time and a second exposure time, wherein the first exposure time and the second exposure time are periodically alternated, the first exposure time is greater than the second exposure time, and the shooting of the camera within the second exposure time is synchronized with the flashing of the LED light ring, thereby ensuring that the camera captures an image when the LED light ring is lit within the second exposure time so as to locate the handle. Among them, during the synchronization process, the head-mounted display device first generates a light strobe signal at a first moment before the camera shoots with a first exposure duration and sends it to the handle to notify the handle to light up the LED light ring. Then the head-mounted display device sends a synchronization signal to the handle at a second moment of the exposure cycle occupied by the second exposure duration. The synchronization signal is received by the handle after a first time interval. In this way, the handle determines the second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to the time information of the received synchronization signal and the time information of the generated light strobe signal, and lights up the LED light ring according to the second time interval, so that the light spot image when the LED light ring is lit, which is collected by the camera within the second exposure duration, is combined with the environmental image and IMU data collected by the camera within the first exposure duration to accurately locate the 6DOf posture of the head-mounted display device and the handle, thereby achieving accurate interaction between the two based on the 6Dof posture.

[0042] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A schematic diagram of the handle button structure provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the interaction between the handle and the head mounted display device provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the structure of a head mounted display device camera and a handle LED light ring provided in an embodiment of the present application;

[0047] Figure 4A schematic diagram of the asynchronous camera shooting and LED light ring flashing provided in an embodiment of the present application;

[0048] Figure 5 The overall architecture diagram of the head mounted display device and the handle synchronous positioning method;

[0049] Figure 6 This is a schematic diagram of a method for setting the first exposure time and the second exposure time;

[0050] Figure 7 A method flow chart of a head mounted display device side in a method for synchronously positioning a head mounted display device and a handle;

[0051] Figure 8 A schematic diagram of synchronization between camera shooting and LED light ring flashing provided in an embodiment of the present application;

[0052] Fig. 9 This is a schematic diagram of the adjustment effect of the first exposure time;

[0053] Fig.10 A method flow chart of a handle side in a method for synchronously positioning a head mounted display device and a handle;

[0054] Fig.11 A flow chart of a method for calculating a second time interval from the generation of a light strobe signal to the lighting of an LED light ring;

[0055] Fig.12 A synchronization schematic diagram of a second exposure duration provided in an embodiment of the present application being greater than the flashing duration of the LED light ring;

[0056] Fig.13 This is a schematic diagram of the interaction process between the head-mounted display device and the handle;

[0057] Fig.14 A flow chart of the interaction method between the head mounted display device and the handle;

[0058] Fig.15 It is a structural schematic diagram of a head mounted display device;

[0059] Fig.16 The structure of the handle provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0061] At present, the positioning and tracking functions of AR / VR and other head-mounted display devices for handles are divided into 3DOF (only posture) positioning and 6DOF (including posture and position) positioning. Due to the higher positioning accuracy of 6DOF, it is the current mainstream interactive control method.

[0062] like Figure 1 As shown, the handle includes multiple operation keys such as trigger key, side grip key, home key, return key, joystick, AB key, etc., and can emit rays. During the interaction process, the ray emitted by the handle points to the application interface of the head-mounted display and forms a cursor on the application interface. In this way, the user operates the control at the focus of the ray cursor through the buttons on the handle to realize the interactive function.

[0063] Take two handles as an example, Figure 2 As shown, it is a schematic diagram of the interaction between the handle and the head-mounted display device, wherein the distance and angle of the left handle and the right handle to the head-mounted display device may be different. When it is necessary to click the "confirm" control on the application interface, one handle can be selected for interactive operation, such as moving the ray cursor emitted by the left handle to the "confirm" control on the application interface. When the user pulls the trigger button of the left handle, the head-mounted display displays the application interface corresponding to the "confirm" control, thereby realizing the interaction with the head-mounted display device. Among them, the user can switch the left and right handles used for interactive operation.

[0064] Generally, the head mounted display device has at least two external cameras, and the handle has a light ring consisting of multiple LED lights flashing at a fixed frequency, such as Figure 3 As shown and not shown, the head mounted display device has a built-in inertial measurement unit (IMU). Similarly, the handle also has a built-in IMU.

[0065] In the interactive scenarios of VR / AR applications, when using the IMU of the dual handles moving in the spatial range and the flashing LED lights to perform real-time positioning and tracking, two problems need to be solved: one is to achieve the simultaneous localization and mapping (SLAM) spatial mobile positioning of the head-mounted display device; the other is to achieve simultaneous positioning and tracking of the dual handles.

[0066] In order to realize the above two functions at the same time, it is necessary to ensure that the camera on the head-mounted display device collects the user environment in real time, and to ensure that the camera's shooting is strictly synchronized with the flashing of the LED light ring on the handle, so as to realize SLAM spatial mobile positioning and dual handles' 6Dof fast and accurate positioning and tracking.

[0067] However, since the head mounted display device and the left and right handles are independent of each other, when positioning is performed based on the images captured by the camera, when the camera shooting frequency on the head mounted display device is not synchronized with the flashing frequency of the LED light rings on the double handles, such as Figure 4 As shown, within the exposure period T, the camera may not be able to capture the image when the LED light ring is on, resulting in the failure of handle positioning, and the inability to simultaneously achieve the SLAM spatial mobile positioning of the head-mounted display device and the 6Dof positioning tracking of the dual handles, thereby producing an erroneous exchange result.

[0068] In view of this, an embodiment of the present application provides a method for synchronous positioning of a head-mounted display device and a handle, which adjusts the shooting mode of the camera and adopts a method combining long exposure time and short exposure time, wherein the image captured within the long exposure time is used for SLAM spatial movement positioning of the head-mounted display device, and the short exposure time is strictly synchronized with the flashing time of the LED light rings on the dual handles, ensuring that the camera can capture the image when the LED light rings on the dual handles are lit, thereby realizing the SLAM spatial movement positioning of the head-mounted display device and the 6Dof fast positioning and tracking of the dual handles. At the same time, the method also adjusts the long exposure time of the camera according to the actual environment, thereby improving the positioning accuracy, and then improving the speed and accuracy of the interactive response, so that the satisfaction of the user experience is improved.

[0069] See also Figure 5 , is an overall architecture diagram of the head-mounted display device and the handle synchronous positioning method. The handle is provided with an LED light ring, a power supply and a physical function key. The microcontroller unit (MCU) of the handle controls the LED light ring to flash at a fixed frequency through a reset circuit, and the handle and the head-mounted display device communicate through their respective wireless modules. The head-mounted display device uses the environmental image captured by the camera and its own IMU data to calculate its own 6Dof pose through the SLAM algorithm. If the dual handles are to be positioned and tracked in real time, it is necessary to ensure that the camera on the head-mounted display device and the LED light rings of the dual handles are synchronized, that is, the LED light rings on the dual handles flash regularly. The head-mounted display device calculates the 6Dof pose of the handle relative to the head-mounted display device based on the spot image captured by the camera when the LED light rings on the dual handles are lit, combined with the IMU data of the handle and the 6DOf pose of the head-mounted display device, thereby realizing the SLAM spatial mobile positioning of the head-mounted display device and the 6Dof positioning tracking of the handle, and then realizing the interactive control of the handle to the head-mounted display device.

[0070] In the embodiment of the present application, in order to achieve synchronous positioning of the head-mounted display device and the handle, the exposure period T of the camera is set to include different exposure times, which are respectively recorded as the first exposure time and the second exposure time. The first exposure time and the second exposure time are periodically alternating, and the first exposure time is greater than the second exposure time, that is, the bit width of the first exposure time in the exposure period is greater than the bit width of the second exposure time in the exposure period, such as Figure 6 As shown. The image captured by the camera during the first exposure time is used to perform SLAM spatial mobile positioning on the head-mounted display device, and the start time of the second exposure time is strictly synchronized with the lighting time of the LED light ring of the double handles. In this way, the spot image of the LED light ring collected by the camera during the second exposure time can be used to perform 6Dof positioning and tracking on the double handles. The exposure period of the camera is determined according to the shooting frame rate of the next international.

[0071] Since the left handle and the right handle are independent of the head mounted display device, the left handle and the right handle are no longer distinguished when describing the synchronous positioning method of the head mounted display device and the handle. The process on the head mounted display device side in this method is as follows: Figure 7 As shown, it mainly includes the following steps:

[0072] S701: At a first moment before the camera takes a picture with a first exposure time, a light strobe signal is generated and sent to the handle.

[0073] Due to the limited volume of the handle, the power supply of the handle is usually small. In order to reduce the power consumption of the handle, the LED light ring is usually intermittently lit at a certain frequency, that is, flashing. Therefore, in order to ensure the synchronous positioning of the head-mounted display device and the handle, at the first moment before the camera of the head-mounted display device shoots with the first exposure time, a strobe signal (Strobe) is generated and sent to the handle to notify the handle to light up the LED light ring.

[0074] S702: sending a synchronization signal to the handle at a fixed time according to the exposure period occupied by the second exposure duration.

[0075] In order to ensure that the handle can be tracked by 6DOf positioning, the second exposure time of the camera needs to be synchronized with the flashing of the LED light ring. Therefore, the head-mounted display device sends a synchronization signal (Sync) to the handle at a fixed time at the second moment. Among them, the regular sending cycle of the synchronization signal is the same as the exposure cycle occupied by the second exposure time, and starting from the second moment, it is received by the handle after the first time interval, that is, the transmission duration of the synchronization signal is the first time interval. In this way, the handle can determine the second time interval from the generation of the light strobe signal to the lighting of the LED light ring based on the time information of the received synchronization signal and the time information of the generated light strobe signal, and light up the LED light ring according to the second time interval, thereby ensuring that the camera can capture the light spot image when the LED light ring is lit every time within the second exposure time.

[0076] The calculation process of the second time interval refers to the description on the handle side and will not be repeated here.

[0077] Optionally, the second moment of sending the Sync signal is the starting moment of the first exposure duration.

[0078] like Figure 8 The figure shows a schematic diagram of the synchronization result of the second exposure time of the camera and the flashing of the LED light ring. At the start moment of the second exposure time, the LED light ring starts to light up, thereby ensuring that the camera can capture the light spot image when the LED light ring is lit at the second exposure moment.

[0079] S703: According to the exposure period occupied by the first exposure duration, the environment image captured by the camera is obtained, and the first 6DOf posture of the handle is determined based on the environment image and the built-in IMU data. In addition, according to the exposure period occupied by the second exposure duration, the light spot image of the LED light ring captured by the camera when it is lit at a second time interval is obtained, and the second 6DOf posture of the handle is determined based on the light spot image, the IMU data of the handle and the first 6DOf posture.

[0080] Since the head-mounted display device has a built-in IMU, the three-axis acceleration and three-axis angular velocity of the head-mounted display device can be collected. By pre-integrating the data collected by the IMU, the 6Dof posture of the head-mounted display device itself can be obtained in real time. However, there is a cumulative error in the IMU prediction. The longer the time, the greater the cumulative error. Therefore, the current posture of the head-mounted display device can be calculated through the environmental image collected by the camera during the first exposure time through a computer vision algorithm, and the predicted 6Dof posture can be optimized through the current posture, so as to obtain the final first 6Dof posture of the head-mounted display device. Similarly, the handle also has a built-in IMU, which can collect the three-axis acceleration and three-axis angular velocity of the handle, and transmit the data collected by the IMU to the head-mounted display device. The head-mounted display device pre-divides the IMU data of the handle and calculates the second 6Dof posture of the handle based on the light spot image when the LED light ring is lit during the second exposure time. Combined with its own 6Dof posture, the 6Dof posture of the handle is also unified to the camera coordinate system, thereby achieving synchronous positioning of the handle and the head-mounted display device.

[0081] In one embodiment, taking into account changes in the camera acquisition environment, the first exposure time of the camera can also be adjusted during the positioning process, that is, the first exposure time in different exposure cycles may be the same or may be different, thereby reducing the impact of environmental changes on feature points in the image, thereby improving the overall positioning accuracy.

[0082] In a specific implementation, an abnormality detection is performed on the environment image captured by the camera during the current first exposure duration, including but not limited to whether the current environment image is overexposed, whether the offset from the previous frame environment image is greater than a preset threshold, and whether there is a deviation in the current environment. When at least one of the above problems is detected, the first exposure duration in the subsequent exposure cycle is increased or decreased according to the detection result within the duration range corresponding to the camera's acquisition frame rate.

[0083] For example, assuming that the acquisition frame rate of the camera is 25 Hz, that is, the exposure period of the camera is T = 4 milliseconds (ms). At this time, the duration range corresponding to the first exposure duration is 2 to 4 ms. Assuming that the first exposure duration of the current environment image is 2 ms, when it is detected that the current environment image has an underexposure problem, the first exposure duration in the next exposure period can be adjusted to 3 ms. Fig. 9 shown.

[0084] It should be noted that the embodiments of the present application do not impose restrictive requirements on the anomaly detection algorithm for environmental images. A deep learning algorithm based on a neural network may be used, or a traditional image processing algorithm may be used.

[0085] In the synchronous positioning method of the head mounted display device and the handle in the embodiment of the present application, the process on the handle side is as follows Fig.10 As shown, it mainly includes the following steps:

[0086] S1001: Receive a light strobe signal generated by a head mounted display device at a first moment before a camera takes a picture with a first exposure duration.

[0087] Since the LED light ring on the handle lights up in a flashing manner, the head-mounted display device generates a light strobe signal and sends it to the handle at the first moment before the camera shoots with the first exposure duration. The handle lights up the LED light ring according to the light strobe signal. However, since the flashing moment of the LED light ring may not be synchronized with the second exposure duration of the camera, the flashing frequency of the LED light ring may be different from the exposure frequency of the camera. It is necessary to synchronize the second exposure duration of the camera with the flashing of the LED light ring. This requires calculating the time interval for the synchronization of the two, that is, starting from the generation of the light strobe signal of the head-mounted display device, the LED light ring of the handle lights up after this time interval. In this way, the light spot image of the LED light ring is captured just within the second exposure duration of the camera, thereby locating the 6DOf posture of the handle.

[0088] S1002: Receive a synchronization signal periodically sent by the head mounted display device at a second moment according to an exposure period occupied by a second exposure duration.

[0089] When the camera captures images with the second exposure duration during the exposure cycle, in order to ensure that the spot image when the LED spot is on is captured, the handle will periodically receive the synchronization signal sent by the head-mounted display device to control the LED light ring to synchronize with the second exposure duration of the camera. The synchronization result is as follows: Figure 8 As shown. The synchronization signal is sent by the head mounted display device at the second moment, and is received by the handle after the first time interval, and the timing sending cycle of the synchronization signal is the exposure cycle occupied by the second exposure duration, thereby ensuring that the camera can capture the light spot image when the LED light ring is lit each time it is exposed with the second duration.

[0090] S1003: Determine a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to the time information of the synchronization signal and the time information of the light strobe signal.

[0091] In specific implementation, the calculation process of the second time interval is as follows: Fig.11 As shown, it mainly includes the following steps:

[0092] S1003_1: Calculate the signal time difference between the second time of sending the synchronization signal and the first time of generating the light strobe signal.

[0093] S1003_2: The difference between the exposure period, the signal time difference and the first time interval is used as the second time interval from the generation of the light strobe signal to the lighting of the LED light ring. The formula of the second time interval is as follows:

[0094] Δt=T-(t2-t1)-ΔT Formula 1

[0095] Wherein, T represents the exposure time, t2 represents the second time instant of sending the synchronization signal, t1 represents the first time instant of generating the light strobe signal, ΔT represents the first time interval from sending to receiving the synchronization signal, and Δt represents the second time interval from generating the light strobe signal to lighting the LED light ring.

[0096] S1004: Light up the LED light ring at a second time interval within the second exposure duration, so that the head-mounted display device determines its own first 6DOf posture according to the environmental image collected by the camera during the exposure period occupied by the first exposure duration and the built-in IMU data, and determines the second 6DOf posture of the handle according to the light spot image when the LED light ring is lit, collected by the camera during the exposure period occupied by the second exposure duration, the IMU data of the handle, and the first 6Dof posture.

[0097] The pose calculation process can be found in the description of the head mounted display device measurement, which will not be repeated here.

[0098] In one embodiment, in order to reduce the amount of calculation on the handle side, the calculation of the second time interval can be performed by the head-mounted display device, and after the second time interval is calculated, it can be notified to the handle so that the handle lights up the LED light ring according to the second time interval.

[0099] In the embodiment of the present application, in order to ensure the synchronous shooting effect of the camera and the LED light ring, the second exposure time of the camera is greater than or equal to the lighting time of the LED light.

[0100] In one example, when the second exposure time of the camera is longer than the lighting time of the LED light, the lighting time of the LED light ring is located in the middle part of the second exposure time of the camera, such as Fig.12 As shown, at this time, the formula of the second time interval is expressed as follows:

[0101] Δt=T-(t2-t1)-ΔT-(S1-S2) / 2 Formula 2

[0102] Wherein, S1 represents the second exposure time of the camera, and S2 represents the lighting time of the LED light ring.

[0103] For example, assuming that the second exposure time of the camera is 500us, and the lighting time of the LED light ring is 100us, it can be known from Formula 2 that the LED light ring starts to flash after the camera is exposed for 200us in the second exposure time.

[0104] In the synchronous positioning method of the head mounted display and the handle provided in the embodiment of the present application, the interaction process between the head mounted display device and the handle is as follows: Fig.13 As shown, first, the head-mounted display device sends a strobe signal (Strobe) to the two handles respectively at a first moment before the camera shoots with a first exposure duration to notify the two handles to light up the LED light ring. Then, at a second moment within the exposure cycle occupied by the second exposure duration, the two handles periodically send a synchronization signal (Sync) to the two handles, so that the two handles calculate the time interval from the head-mounted display device generating the Strobe to lighting up the LED light ring according to the time information of the Sync and the time information of the Strobe, so as to keep the flashing of the LED light ring synchronized with the second short exposure duration, thereby realizing the synchronous positioning of the head-mounted display device and the two handles.

[0105] The method flow of the interaction between the head mounted display device and the handle is as follows Fig.14 As shown, it mainly includes the following steps:

[0106] S1401: The head mounted display device generates a light strobe signal at a first moment before an exposure period occupied by a first exposure duration.

[0107] S1402: The head mounted display device sends an isostroboscopic signal to the dual handles.

[0108] S1403: The head mounted display device sends a synchronization signal to each of the two handles at a second moment of the exposure period occupied by the second exposure duration.

[0109] S1404: After the first time interval, the two handles receive synchronization signals respectively.

[0110] S1405: The two handles respectively calculate the signal time difference between the second moment of sending the synchronization signal and the first moment of generating the light strobe signal.

[0111] S1406: The two handles respectively use the difference between the exposure period, the signal time difference and the first time interval as the second time interval from the generation of the light strobe signal to the lighting of the LED light ring.

[0112] S1407: The two handles light up their respective LED light rings at a second time interval.

[0113] S1408: The head mounted display device uses the camera to collect the environment image within the first exposure time, as well as its own IMU data, and calculates its own first 6DoF posture.

[0114] S1409: The head-mounted display device calculates the second 6DoF posture of each of the two handles by using the light spot images of the LED light rings of each of the two handles when they are lit at the second time interval captured by the camera within the second exposure time, as well as the acquired IMU data of the handles and its own first 6DoF posture.

[0115] In the synchronous positioning method of the head-mounted display device and the handle provided in the embodiment of the present application, different exposure times are set within the exposure cycle of the camera, and a light strobe signal is sent to the handle at a first moment before the exposure cycle occupied by the first exposure time, and a synchronization signal is sent to the handle at a second moment in the exposure cycle occupied by the second exposure time. The handle calculates the second time interval from the generation of the light strobe signal to the lighting of the LED light ring based on the time difference between the two signals, combined with the transmission time (first time interval) of the synchronization signal and the exposure cycle, thereby achieving strict synchronization between the second exposure time of the camera and the flashing of the LED light ring of the handle, thereby ensuring the real-time collection of environmental information by the camera, and satisfying the strict synchronization between the shooting of the camera and the flashing of the LED light ring of the handle, thereby realizing the SLAM spatial positioning of the head-mounted display device and the 6Dof fast and accurate positioning and tracking of the dual handles, thereby ensuring the fast and accurate control of the head-mounted display device by the dual handles, and improving the interactive response results and user experience.

[0116] Based on the same technical concept, an embodiment of the present application provides a head-mounted display device, which can be a virtual reality device or an augmented reality device. The head-mounted display device can implement the steps of the above-mentioned method for synchronous positioning of a head-mounted display device and a handle, and can achieve the same technical effect.

[0117] See also Fig.15 The head-mounted display device includes an external camera 1501 provided with a first exposure time and a second exposure time, the first exposure time and the second exposure time are periodically alternated, the first exposure time is greater than the second exposure time, the head-mounted display device has a built-in processor 1502, a memory 1503, a display 1504, an IMU 1505 and a communication interface 1506, the camera 1501, the communication interface 1506, the IMU 1505, the display 1504, the memory 1503 and the processor 1502 are connected via a bus 1507;

[0118] The head mounted display device communicates with a handle provided with a flashing LED light ring and an IMU through the communication interface 1506, so that the handle controls the application interface displayed on the display 1504;

[0119] The memory 1503 stores a computer program, and the processor 1502 performs the following operations according to the computer program:

[0120] At a first moment before the camera 1501 takes a photo with the first exposure duration, a light strobe signal is generated and sent to the handle;

[0121] According to the exposure cycle occupied by the second exposure duration, a synchronization signal is periodically sent to the handle at a second moment, and the synchronization signal is received by the handle within a first time interval at the second moment, so that the handle determines a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to time information of the received synchronization signal and time information of the generated light strobe signal, and lights up the LED light ring according to the second time interval;

[0122] According to the exposure period occupied by the first exposure time, the environment image captured by the camera 1501 is obtained, and according to the environment image and the built-in IMU data, its own first 6DOf pose is determined; and, according to the exposure period occupied by the second exposure time, the light spot image when the LED light ring is lit captured by the camera 1501 is obtained, and the second 6DOf pose of the handle is determined according to the light spot image, the IMU data of the handle and the first 6DOf pose.

[0123] Optionally, after acquiring the environment image captured by the camera within the first exposure duration, the processor 1502 further executes:

[0124] Performing anomaly detection on the environment image acquired by the camera during the current first exposure duration, wherein the anomaly detection includes at least one of overexposure, offset, and deviation;

[0125] When at least one abnormality is detected, within the time range corresponding to the acquisition frame rate of the camera 1501, the first exposure time in the subsequent exposure cycle is adjusted to increase or decrease according to the detection result.

[0126] Optionally, the second exposure duration is greater than or equal to the lighting duration of the LED light.

[0127] Optionally, there are two handles.

[0128] It should be noted that Fig.15 It is only an example to provide the necessary hardware for the head mounted display device to execute the steps of the method for synchronous positioning of a head mounted display device and a handle provided in an embodiment of the present application. If not shown, the head mounted display device may also include conventional hardware such as a speaker, a pickup, left and right eyeglasses, and a power supply.

[0129] Based on the same technical concept, an embodiment of the present application provides a handle that can implement the steps of the above-mentioned method for synchronous positioning of a head-mounted display device and a handle, and can achieve the same technical effect.

[0130] See also Fig.16 The handle is provided with a flashing LED light ring 1601, and the handle is provided with a processor 1602, a memory 1603, a communication interface 1604 and an IMU 1605. The LED light ring 1601, the IMU 1605, the communication interface 1604, the memory 1603 and the processor 1602 are connected via a bus 1606;

[0131] The handle communicates with the head-mounted display device through the communication interface 1604 to control the application interface displayed on the display of the head-mounted display device, wherein the camera on the head-mounted display device is set with a first exposure time and a second exposure time, the first exposure time and the second exposure time are periodically alternated, and the first exposure time is greater than the second exposure time;

[0132] The memory 1603 stores a computer program, and the processor 1602 performs the following operations according to the computer program:

[0133] receiving a light strobe signal generated by the head mounted display device at a first moment before the camera takes a picture with the first exposure duration;

[0134] receiving, according to the exposure period occupied by the second exposure duration, a synchronization signal regularly sent by the head mounted display device at a second moment, wherein a first time interval has elapsed from a time when the synchronization signal is received to a time when the second moment is received;

[0135] Determine a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to the time information of the synchronization signal and the time information of the light strobe signal;

[0136] Light up the LED light ring 1601 at the second time interval, so that the head-mounted display device determines its own first 6DOf posture according to the environmental image collected by the camera during the exposure period occupied by the first exposure time and the built-in IMU data, and determines the second 6DOf posture of the handle according to the light spot image when the LED light ring is lit, collected by the camera during the exposure period occupied by the second exposure time, the IMU data of the handle, and the first 6Dof posture.

[0137] Optionally, the processor 1602 determines a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to the time information of the synchronization signal and the time information of the light strobe signal, and the specific operation is:

[0138] Calculate the signal time difference between the second time of sending the synchronization signal and the first time of generating the light strobe signal;

[0139] The difference between the exposure period, the signal time difference and the first time interval is used as the second time interval from the generation of the light strobe signal to the lighting of the LED light ring.

[0140] Optionally, the second exposure duration is greater than or equal to the lighting duration of the LED light.

[0141] Optionally, when the second exposure duration is greater than the lighting duration of the LED light, the lighting duration of the LED light ring is located in the middle part of the second exposure duration, and the formula for the second time interval is expressed as:

[0142] Δt=T-(t2-t1)-ΔT-(S1-S2) / 2

[0143] Wherein, T represents the exposure duration, t2 represents the second moment of sending the synchronization signal, t1 represents the first moment of generating the light strobe signal, ΔT represents the first time interval from sending to receiving the synchronization signal, S1 represents the second exposure duration, S2 represents the lighting duration of the LED light ring, and Δt represents the second time interval from the generation of the light strobe signal to the lighting of the LED light ring.

[0144] in, Fig.15 and Fig.16 The memory in the term may be a volatile memory, such as a random-access memory (RAM); the memory may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories;

[0145] The processor may include one or more central processing units (CPU) or a digital processing unit, etc.

[0146] It should be noted that Fig.16 This is just an example, and provides the necessary hardware for the handle to execute the steps of the synchronous positioning method of a head mounted display device and a handle provided in an embodiment of the present application. If not shown, the handle may also include conventional hardware such as physical buttons and a power supply.

[0147] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, a method for synchronous positioning of a head-mounted display device and a handle in the aforementioned embodiment can be completed.

[0148] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a method for synchronous positioning of a head-mounted display device and a handle in the aforementioned embodiment.

[0149] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for synchronous positioning of a head mounted display and a handle, characterized in that: Applied to a head-mounted display device, a camera on the head-mounted display device is provided with a first exposure time and a second exposure time, the first exposure time and the second exposure time are periodically alternated, the first exposure time is greater than the second exposure time, the head-mounted display device communicates with a handle, the handle is provided with a flashing LED light ring, and the method comprises: At a first moment before the camera takes a photo with the first exposure duration, a light strobe signal is generated and sent to the handle; According to the exposure cycle occupied by the second exposure duration, a synchronization signal is periodically sent to the handle at a second moment, and the synchronization signal is received by the handle within a first time interval at the second moment, so that the handle determines a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to time information of the received synchronization signal and time information of the generated light strobe signal, and lights up the LED light ring according to the second time interval; According to the exposure period occupied by the first exposure time, the environment image captured by the camera is obtained, and according to the environment image and the built-in IMU data, the first 6DOf posture of the handle is determined. In addition, according to the exposure period occupied by the second exposure time, the light spot image when the LED light ring is lit is obtained, and the second 6DOf posture of the handle is determined according to the light spot image, the IMU data of the handle and the first 6DOf posture.

2. The method according to claim 1, characterized in that After acquiring the environment image captured by the camera within the first exposure time, the method includes: Performing anomaly detection on the environment image acquired by the camera during the current first exposure duration, wherein the anomaly detection includes at least one of overexposure, offset, and deviation; When at least one abnormality is detected, within the time range corresponding to the acquisition frame rate of the camera, the first exposure time in the subsequent exposure cycle is adjusted to increase or decrease according to the detection result.

3. The method according to claim 1 or 2, characterized in that The second exposure duration is greater than or equal to the lighting duration of the LED light.

4. The method according to claim 1 or 2, characterized in that: The number of the handles is two.

5. A method for synchronous positioning of a head mounted display device and a handle, characterized in that: Applied to a handle, the handle is provided with a flashing LED light ring, the handle communicates with a head-mounted display device, a camera on the head-mounted display device is provided with a first exposure time and a second exposure time, the first exposure time and the second exposure time are periodically alternated, and the first exposure time is greater than the second exposure time, the method comprising: receiving a light strobe signal generated by the head mounted display device at a first moment before the camera takes a picture with the first exposure duration; receiving, according to the exposure period occupied by the second exposure duration, a synchronization signal regularly sent by the head mounted display device at a second moment, wherein a first time interval has elapsed from a time when the synchronization signal is received to a time when the second moment is received; Determine a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to the time information of the synchronization signal and the time information of the light strobe signal; Light up the LED light ring at the second time interval so that the head-mounted display device determines its own first 6DOf posture according to the environmental image collected by the camera during the exposure period occupied by the first exposure time and the built-in IMU data, and determines the second 6DOf posture of the handle according to the light spot image when the LED light ring is lit, collected by the camera during the exposure period occupied by the second exposure time, the IMU data of the handle, and the first 6Dof posture.

6. The method according to claim 5, characterized in that The determining, according to the time information of the synchronization signal and the time information of the light strobe signal, a second time interval from the generation of the light strobe signal to the lighting of the LED light ring comprises: Calculate the signal time difference between the second time of sending the synchronization signal and the first time of generating the light strobe signal; The difference between the exposure period, the signal time difference and the first time interval is used as the second time interval from the generation of the light strobe signal to the lighting of the LED light ring.

7. The method according to claim 5 or 6, characterized in that The second exposure duration is greater than or equal to the lighting duration of the LED light.

8. The method according to claim 7, characterized in that When the second exposure time is greater than the lighting time of the LED light, the lighting time of the LED light ring is located in the middle of the second exposure time, and the formula of the second time interval is expressed as: Δt=T-(t2-t1)-ΔT-(S1-S2) / 2 Wherein, T represents the exposure duration, t2 represents the second moment of sending the synchronization signal, t1 represents the first moment of generating the light strobe signal, ΔT represents the first time interval from sending to receiving the synchronization signal, S1 represents the second exposure duration, S2 represents the lighting duration of the LED light ring, and Δt represents the second time interval from the generation of the light strobe signal to the lighting of the LED light ring.

9. A head mounted display device, characterized in that: The external camera of the head-mounted display device is set with a first exposure time and a second exposure time, the first exposure time and the second exposure time are periodically alternated, the first exposure time is greater than the second exposure time, the head-mounted display device is built-in with a processor, a memory, a display, an IMU and a communication interface, and the communication interface, the IMU, the display, the memory and the processor are connected via a bus; The head mounted display device communicates with a handle provided with a flashing LED light ring and an IMU through the communication interface, so that the handle controls the application interface displayed on the display; The memory stores a computer program, and the processor performs the following operations according to the computer program: At a first moment before the camera takes a photo with the first exposure duration, a light strobe signal is generated and sent to the handle; According to the exposure cycle occupied by the second exposure duration, a synchronization signal is periodically sent to the handle at a second moment, and the synchronization signal is received by the handle within a first time interval at the second moment, so that the handle determines a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to time information of the received synchronization signal and time information of the generated light strobe signal, and lights up the LED light ring according to the second time interval; According to the exposure period occupied by the first exposure time, the environment image captured by the camera is obtained, and according to the environment image and the built-in IMU data, the first 6DOf posture of the handle is determined. In addition, according to the exposure period occupied by the second exposure time, the light spot image when the LED light ring is lit is obtained, and the second 6DOf posture of the handle is determined according to the light spot image, the IMU data of the handle and the first 6DOf posture.

10. A handle, characterized in that: The handle is provided with a flashing LED light ring, and the handle is provided with a processor, a memory, a communication interface and an IMU built in, and the IMU, the communication interface, the memory and the processor are connected via a bus; The handle communicates with the head-mounted display device through the communication interface to control the application interface displayed on the display of the head-mounted display device, wherein the camera on the head-mounted display device is set with a first exposure time and a second exposure time, the first exposure time and the second exposure time are periodically alternated, and the first exposure time is greater than the second exposure time; The memory stores a computer program, and the processor performs the following operations according to the computer program: receiving a light strobe signal generated by the head mounted display device at a first moment before the camera takes a picture with the first exposure duration; receiving, according to the exposure period occupied by the second exposure duration, a synchronization signal regularly sent by the head mounted display device at a second moment, wherein a first time interval has elapsed from a time when the synchronization signal is received to a time when the second moment is received; Determine a second time interval from the generation of the light strobe signal to the lighting of the LED light ring according to the time information of the synchronization signal and the time information of the light strobe signal; Light up the LED light ring at the second time interval so that the head-mounted display device determines its own first 6DOf posture according to the environmental image collected by the camera during the exposure period occupied by the first exposure time and the built-in IMU data, and determines the second 6DOf posture of the handle according to the light spot image when the LED light ring is lit, collected by the camera during the exposure period occupied by the second exposure time, the IMU data of the handle, and the first 6Dof posture.

Images